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VIDEO DATA MULTIPLEXING DEVICE, VIDEO DATA MULTIPLEXING 
CONTROL METHOD, ENCODED STREAM MULTIPLEXING DEVICE 
AND METHOD, AND ENCODING DEVICE AND METHOD 



Technical Field 

The present invention relates to a video data multiplexing device 
and a video data multiplexing control method for compressing and encoding 
a plurality of program data including video data and multiplexing them, an 
encoded stream multiplexing device and method for multiplexing encoded 
streams, and an encoding device and method for encoding video data. 
Background Art 

Recently, digital broadcasting in which image data and so on are 
transmitted and received as digital data is drawing the attention. An 
advantage of the digital broadcasting is that more program data (hereinafter 
referred to as programs) can be transmitted over the same transmission 
channel as compared with analog broadcasting. This is due in large part to 
the fact that image data can be compressed before being transmitted . As 
the method for compressing image data, there is, for example, a bidirectional 
predictive encoding system adopted in MPEG (Moving Picture Experts 
Group) standards. In this bidirectional predictive encoding system, three 
types of encoding, i.e., intra-frame encoding, inter-frame forward predictive 
encoding, and bidirectional predictive encoding are performed. Images 
obtained by using respective encoding types are called I picture (intra 
encoded picture), P picture (predictive encoded picture), and B picture 
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(bidirectionally predictive encoded picture), respectively. 

FIG. 1 is a block diagram showing an example of a configuration of a 
digital broadcasting system using an image encoding device of the 
bidirectional predictive encoding system according to the MPEG standards. 
A digital broadcasting system 300 includes a plurality of image encoding 
devices 30 lj through 30 l n (where n is an integer value of minimum 2) for 
compressing and encoding program data such as image data, a multiplexer 
302 connected to these image encoding devices 301.! through 301 n so as to 

P 

fl multiplex data compressed and encoded by the respective image encoding 

devices 30 lj through 301 n , and a modulator 303 for modulating output image 
data multiplexed by the multiplexer 302. In the digital broadcasting 
system 300, image data are compressed and encoded by n image encoding 
fifi devices 30 l x through 30 l n , and outputted to the multiplexer 302. The 

M= multiplexer 302 multiplexes compressed and encoded data inputted from the 

« image encoding devices 30^ through 301 n , and outputs the multiplexed data 

to the modulator 303 with a fixed data rate (speed) of, for example, 
approximately 30 Mbps. The compressed and encoded data supplied from 
the multiplexer 302 is modulated by the modulator 303, and transmitted to 
each home 305 via a broadcasting satellite 304. 

FIG. 2 is a block diagram showing an example of a configuration of 
the image encoding device illustrated in FIG. 1. The image encoding device 
301 (which is representative of 30^ through 301J includes : a subtraction 
circuit 310 for deriving a difference between inputted image data S 101 and 
predicted image data; a DCT circuit 311 for conducting DOT on output data 



P 
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of the subtraction circuit 310 by each DCT block and outputting a DCT 
coefficient; a quantization circuit 312 for quantizing output data of the DCT 
circuit 311; a variable length encoding circuit 313 for conducting variable 
length encoding on output data of the quantization circuit 312; a buffer 
memory 314 for temporarily holding output data of the variable length 
encoding circuit 313 and outputting it as compressed image data S 102 
comprising a bit stream having a fixed bit rate; an inverse quantization 
circuit 315 for conducting inverse quantization on the output data of the 
quantization circuit 312; an inverse DCT circuit 316 for conducting inverse 
% DCT on output data of the inverse qunatization circuit 315; an addition 

Tji circuit 317 for adding output data of the inverse DCT circuit 316 to the 

a predicted image data and outputting a resulting sum; a motion 

a 

CP compensation circuit 318 for holding the output data of the addition circuit 

!h* 317, conducting motion compensation on the output data on the basis of a 

O motion vector, and outputting predicted image data to the subtraction circuit 

310 and the addition circuit 317; a bit rate control unit 319 for controlling a 
target code quantity on the basis of generated bit quantity data S 103 supplied 
from the variable length encoding circuit 313 so as to make the compressed 
image data S 102 outputted from the buffer memory 314 become a bit stream 
having a fixed rate; and a motion detection circuit 309 for detecting the 
motion vector on the basis of the inputted image data S 101 and transmitting 
the detected motion vector to the motion compensation circuit 318. 

In the image encoding device 301 shown in FIG. 2, the image data 
S 101 is inputted to the subtraction circuit 310. The output signal of the 
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subtraction circuit 310 is inputted to the DCT circuit 311, and subjected to 
DCT. The output signal of the DCT circuit 311 is quantized by the 
quantization circuit 312, and subjected to variable length encoding in the 
variable length encoding circuit 313. The output data of the variable length 
encoding circuit 313 is temporarily held by the buffer memory 314 and 
outputted as the compressed image data S 102 . 

Furthermore, from the variable length encoding circuit 313, the 
generated bit quantity data S 103 is outputted to the bit rate control unit 319. 

O 

y3 On the basis of the generated bit quantity data S 103 , the bit rate control unit 



Ly 



319 determines the target code quantity. According to the target code 
quantity, the bit rate control unit 319 controls quantization characteristic in 
the quantization circuit 312. 

By the way, in image compression and encoding in digital 
broadcasting, it is necessary to keep the image quality high while 
suppressing the data quantity being compressed and encoded to be the 
transmission capacity of a transmission channel or less. 

There is a technique called "statistical multiplexing", as a method 
for letting more programs flow through a transmission channel having a 
predetermined transmission capacity. The statistical multiplexing is a 
technique for transmitting a larger number of programs by dynamically 
changing transmission rates of respective programs. According to this 
statistical multiplexing, for example, the transmission rates are lowered on 
programs which transmission rates reduction do not cause a conspicuous 
image quality degradation, so as to make it possible to transmit a larger 
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number of programs. The statistical multiplexing utilizes the fact that such 
portions (time periods) of respective programs in which image quality 
degradation are conspicuous, rarely overlap with one another 
simultaneously. Therefore, when such a portion of a certain program which 
makes conspicuous image quality degradation is transmitted, image quality 
degradation is not conspicuous in other programs in many cases even if the 
code rate is lowered. Accordingly, it is possible to lower code rates of other 
programs and assign the higher code rate to such a program that image 

C3 

'p. quality degradation is conspicuous. By using the statistical multiplexing, a 

J larger number of programs than the ordinary case can thus be transmitted. 

m In such a statistical multiplexing, a bit rate assigning technique for 

= determining a bit rate quantity as code rates to be assigned to respective 

p 

B! programs is an important factor in determining the image quality and so on. 

I* 

According to a representative technique proposed heretofore as a bit rate 

f 

'M assigning technique, quantization steps used in respective programs are 

monitored and they are subjected to feedback control so that the 
quantization steps will become the same in all programs or quantization 
steps will have weights preset for respective programs. In such a feedback 
control, after an image is encoded, the next bit rate is determined on the 
basis of a quantization step used at the time of encoding the image. 
Therefore, when the pattern is abruptly changed to a difficult pattern caused 
by such as scene change, a countermeasure will be delayed, resulting in a 
problem that image distortion occurs significantly. 

On the other hand, in order to solve the problem of such a 



countermeasure delay of the control system due to feedback control, the 
present applicant has proposed a technique called feedforward control, in 
which, encoding difficulty representing the difficulty of the encoding 
regarding an image to be encoded, is derived beforehand, and bit rates of 
respective programs are determined according to this encoding difficulty. 
This feedforward control technique is implemented basically by distributing 
a total bit rate after being multiplexed among respective programs according 
to ratios of encoding difficulty data read from the respective programs 
beforehand. Allocation of bit rates to respective programs according to 
ratios of the encoding difficulty data is determined by proportional allocation 
as represented by the following expression (1). 

R, = (D/ EDJx Total_Rate -(1) 

In the expression (1), Ri means a bit rate of a program number i, D ; 
encoding difficulty of the program per unit time, Total_Rate a total bit rate, 
and £ a sum total over k = 1 to L (where L is the total number of programs). 

Alternatively, allocation of bit rates to respective programs is 
determined as represented by the following expression (2) by adding weight 
coefficients W; to respective programs. 

R^lWjXDi/ZCWkXDk)} xTotal_Rate ...(2) 

In this way, the bit rates are proportionally allocated according to 



the encoding difficulty values. 

FIG. 3 is a block diagram showing an example of a configuration of a 
multiplexing device using the above described statistical multiplexing. This 
multiplexing device 330 includes a plurality of image encoding devices 331 x 
to 331„ for conducting compression encoding on respective inputted programs 
P x to P n , a statistical multiplexing controller 332 connected to n image 
encoding devices 331.! to 331 n to control the image encoding devices 331 x to 
331 n , and a multiplexer 333 for multiplexing compressed and encoded data 
St 2 to St n compressed and encoded by the image encoding devices 331.! to 331 n . 
The image encoding devices 331 x to 331 n in advance derive encoding 
difficulties D x to D n each of which represents the difficulty of encoding with 
regard to an image to be encoded, and output the encoding difficulties D x to 
D n to the statistical multiplexing controller 332. By distributing the total 
bit rate after being multiplexed according to the ratios of these encoding 
difficulties B 1 to D n , the statistical multiplexing controller 332 determines 
target bit rates for the respective programs P x to P n according to the ratios of 
the encoding difficulties Dj to D n> and outputs control data CI^ to CR n such as 
the target bit rates to the image encoding devices 331 x to 331 n , respectively. 
On the basis of the control data to CP^ such as the target bit rates 
supplied from the statistical multiplexing controller 332, each of the image 
encoding devices 331 x to 331 n conducts compression encoding on the 
programs P x to P n , and outputs compressed and encoded data St! to St n to the 
multiplexer 333. The multiplexer 333 multiplexes the respective 
compressed and encoded data St x to St n which were inputted, generating 
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image data Sm to be outputted, and outputs the image data Sm to the 
modulator 303 shown in FIG. 1. 

FIG. 4 is a block diagram showing an example of a configuration of 
the image encoding devices illustrated in FIG. 3. In this image encoding 
device 331 (which is representative of 331! to 331^, the same components as 
those of the image encoding device 301 are denoted by same marks, and 
description thereof will be omitted. In this image encoding device 331, the 
statistical multiplexing controller 332 controls the target code quantity, 

er 

V instead of the bit rate control unit 319 of the image encoding device 301 

yj 

\. shown in FIG. 2. The motion detection circuit 309 in this image encoding 

■M 

m 

'if, device 331 outputs a ME residual to the statistical multiplexing controller 

s 332 as encoding difficulties D while deriving a motion vector. In brief, the 

□ 

m ME residual is the sum total of absolute values or the sum total of squared 

iM= values of motion prediction errors over a whole picture. On the basis of the 

* encoding difficulties D supplied from the motion detection circuits 309 of 

each of the image encoding devices 331, the statistical multiplexing 
controller 332 conducts control using statistical multiplexing, generates 
control data CR such as target code quantities, and outputs the control data 
CR to quantization circuits 342. On the basis of the control data CR, the 
quantization circuit 342 quantizes data outputted from the DCT circuit 311, 
and outputs the quantized data to a variable length encoding circuit 343. 
By the way, the statistical multiplexing controller 332 is supplied with the 
encoding difficulties D from each of the image encoding devices 331, and 
outputs the control data CR to the quantization circuits 342 each of the 
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image encoding devices 331. In FIG. 4, D is representative of D x to D n , and 
CR is representative of CR X to CR n . 

Each of FIGS. 5A to 5C shows an example of a bit rate change in 
each of the image encoding devices included in the multiplexing device using 
the statistical multiplexing. FIG. 5A shows a bit rate change in the image 
encoding device 331!. FIG. 5B shows a bit rate change in the image 
encoding device 331 2 . FIG. 5C shows a bit rate change in the image 
encoding device 33 l n . The axis of ordinates represents the bit rate of the 



?*? image encoding device, and the axis of abscissas represents time. As 

m 

\p described above, the statistical multiplexing utilizes the fact that such 

m 

m portions (time periods) of respective programs in which image quality 

¥ degradation is conspicuous rarely overlap with one another simultaneously, 

ffl Therefore, when such a portion of a certain program in which image quality 

H= degradation is conspicuous is transmitted, image quality degradation is not 

*B conspicuous in other programs even if the bit rate is lowered. Accordingly, 

it is possible to assign some of bit rates of the other programs to such a 
program that image quality degradation is conspicuous. 

As shown in FIGs. 5A to 5C, bit rates assigned to respective image 
encoding devices 331 x to 331 n in which respective programs are inputted are 
controlled so as to become variable rates in the time axis direction. As 
shown in FIG. 5A, for example, the bit rate of the image encoding device 33 1! 
is high at, time A. This is because in the image encoding device 331 x , the 
value of the encoding difficulty becomes high at the time A as the image 
motion is rapid or the pattern is complicated then. Therefore, a high bit 
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rate is assigned to the image encoding device 33 l x at the time A. On the 
other hand, in the image encoding device 331 n at time B, the encoding 
difficulty value becomes low as shown in FIG. 5C, because the image is 
nearly still or the pattern is simple. Therefore, a low bit rate is assigned to 
the image encoding device 33 l n at the time B. Furthermore, the sum total 
of the bit rates assigned to respective image encoding devices 331j to 331 n 
must be fixed. For example, at time C of FIGS. 5A to 5C, the sum total of 
bit rates Rl to Rn respectively assigned to each image encoding devices 33 l x 
to 331 n is a fixed quantity, and this becomes the bit rate of the modulator 303 
shown in FIG. 1. By using the statistical multiplexing, a larger number of 
programs than the ordinary case can thus be transmitted. 

However, in the multiplexing device 330 shown in FIG. 4, the 
encoding difficulties D x to D n are sent from the respective image encoding 
devices 331! to 331 n to the statistical multiplexing controller 332. The 
control data CRj to CR n such as target bit rates derived from the statistical 
multiplexing controller 332 on the basis of those encoding difficulties D x to D n 
are sent to the image encoding devices 33^ to 331 n , respectively. Therefore, 
in the statistical multiplexing controller, an input portion and an output 
portion corresponding to the image encoding devices 33 lj to 33 l n become 
necessary, resulting in the enlargement of the scale and a problem that data 
exchange with respective image encoding devices become complicated. 

Furthermore, as a multiplexing device using statistical multiplexing, 
the present applicant previously has proposed a multiplexing device 
(Japanese Patent Application No. 9-179882), wherein the statistical 



multiplexing computer conducting bit rate control of feedforward type using 
a computer for general purpose instead of a statistical multiplexing 
controller serving as a dedicated device is connected to respective image 
encoding devices via a network such as Ethernet, and exchange of the 
encoding difficulties between respective image encoding devices and the 
statistical multiplexing computer is conducted via the Ethernet. FIG. 6 is a 
block diagram showing an example of a configuration using the statistical 
multiplexing computer. In this statistical multiplexing system 400, 
respective image encoding devices 402 x to 402n output transport streams St x 
to St n each containing an encoded data sequence corresponding to one 
channel, to a multiplexer 404. The image encoding devices 402 x to 402n 
output encoding difficulties to D n respectively, for controlling compression 
encoding in the image encoding devices 402 x to 402n, to the statistical 
multiplexing computer 403. By taking a packet as a unit, the encoding 
difficulties D x to D n are sent from the image encoding devices 402 , to 402n to 
the statistical multiplexing computer 403 via an Ethernet 405, respectively. 
Target bit rates Rate x to Rate n respectively assigned to the encoding 
difficulties D x to D n are returned to the respective image encoding devices 
402 x to 402n via the same Ethernet 405. 

According to the statistical multiplexing system shown in FIG. 6, it 
thus becomes possible to transmit the encoding difficulties D x to D n and the 
target bit rates Rat ei to Rate n efficiently between the image encoding devices 
402 x to 402 n and the statistical multiplexing computer 403. 

However, in the statistical multiplexing system shown in FIG. 6, the 
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network serving as transmission channels between the image encoding 
devices 402 x to 402 n and the statistical multiplexing computer 403, such as 
the Ethernet 405, goes down in efficiency when the number of transmitted 
packets increases. Thus there is a possibility that a trouble may occur in 
the statistical multiplexing system 400 for controlling a large number of 
image encoding devices 402 1 to 402 n connected to the network. 

Furthermore, the Ethernet 405 is usually used for control systems 
other than the statistical multiplexing system as well. When a command is 
sent for the control, there is a possibility that the transmitting of the 
encoding difficulties D x to D n and the target bit rates Ratej to Rate n are 
affected. 

Disclosure of Invention 

In view of such problems, the present invention has been performed. 
An object of the present invention is to provide a video data multiplexing 
device, a video data multiplexing control method, an encoded stream 
multiplexing device, and method, and an encoding device, and method, which 
make it possible to efficiently transmit data for statistical multiplexing 
required for control using statistical multiplexing. 

A video data multiplexing device of the present invention comprises: 
a plurality of encoding means for encoding program data respectively 
including video data, outputting resultant encoded streams, while 
generating statistical multiplexing data required for control using statistical 
multiplexing, and outputting the generated data on the same transmission 
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channels as the encoded streams; multiplexing means for acquiring the 
encoded streams and the statistical multiplexing data from the respective 
encoding means via the transmission channels, and multiplexing and 
outputting them; and encoding control means for acquiring the statistical 
multiplexing data of respective encoding means from the output of the 
multiplexing means, and conducting control using statistical multiplexing on 
respective encoding means on the basis of the statistical multiplexing data. 

In the video data multiplexing device of the present invention, by 
the respective encoding means, each of program data is encoded, statistical 
multiplexing data required for control using statistical multiplexing, are 
generated, and the data are outputted to the same transmission channels as 
the encoded streams are transmitted. Also, by the multiplexing means, the 
encoded streams and the statistical multiplexing data from the respective 
encoding means are acquired via the transmission channels, and they are 
multiplexed and outputted. Furthermore, by the encoding control means, 
the statistical multiplexing data of respective encoding means from the 
output of the multiplexing means are acquired, and control using statistical 
multiplexing on respective encoding means in conducted on the basis of the 
statistical multiplexing data. 

A video data multiplexing control method of the present invention 
for effecting control on the respective encoding means by the encoding 
control means, used in a video data multiplexing device including a plurality 
of encoding means for encoding program data respectively including video 
data and outputting encoded streams, multiplexing means for multiplexing 
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the encoded streams outputted from the respective encoding means, and 
encoding control means for controlling the respective encoding means, 
comprises: a statistical multiplexing data output procedure in the encoding 
means, generating statistical multiplexing data required for control using 
statistical multiplexing and outputting the generated data to the same 
transmission channels as the encoded streams are transmitted; a 
multiplexing procedure in the multiplexing means, acquiring the encoded 
streams and the statistical multiplexing data from the respective encoding 
means via the transmission channels, and multiplexing and outputting 
them; and an encoding control procedure in the encoding control means for 
acquiring the statistical multiplexing data of the respective encoding means 
from output of the multiplexing means, and conducting control using 
statistical multiplexing on the respective encoding means on the basis of the 
statistical multiplexing data. 

In the video data multiplexing control method of the present 
invention, statistical multiplexing data required for control using statistical 
multiplexing is generated and outputted on the same transmission channels 
as the encoded streams are transmitted by the statistical multiplexing data 
output procedure in the encoding means. The encoded streams and the 
statistical multiplexing data are acquired from the respective encoding 
means via the transmission channels, and multiplexed and outputted by the 
multiplexing procedure in the multiplexing means. The statistical 
multiplexing data of the respective encoding means is acquired from output 
of the multiplexing means, and control using statistical multiplexing is 
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conducted on the respective encoding means on the hasis of the statistical 
multiplexing data, by the encoding control procedure in the encoding control 
means. 

Another video data multiplexing device of the present invention 
comprises: a plurality of encoding means for encoding program data 
respectively including video data, outputting a resultant encoded streams, 
while generating statistical multiplexing data required for control using 
statistical multiplexing, and outputting the generated data on the same 
transmission channel as the encoded streams are transmitted; multiplexing 
means for acquiring the encoded streams and the statistical multiplexing 
data from the respective encoding means via the transmission channels, 
conducting multiplexing processing on the encoded streams and the 
statistical multiplexing data at a first rate greater than a data transmission 
rate on a transmission channel of a subsequent stage, outputting first data 
including the statistical multiplexing data, while conducting multiplexing 
processing on data, obtained by removing the statistical multiplexing data 
from the data outputted from the respective encoding means, at a second 
rate equal to a data transmission rate on the transmission channel of the 
subsequent stage, and outputting second data, which does not include the 
statistical multiplexing data, to the transmission channel of the subsequent 
stage; and encoding control means for acquiring the statistical multiplexing 
data of the respective encoding means from the first data outputted from the 
multiplexing means, and conducting control using statistical multiplexing on 
the respective encoding means on the basis of the statistical multiplexing 
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data. 

In another video data multiplexing device of the present invention, 
the respective encoding means code respective program data and output 
resultant encoded streams. In addition, the respective encoding means 
generate statistical multiplexing data required for control using statistical 
multiplexing, and output the generated data on the same transmission 
channels as the encoded streams are transmitted. Furthermore, the 
multiplexing means acquires the encoded stream and the statistical 
multiplexing data from the respective encoding means via the transmission 
channels, conducts multiplexing processing on the encoded streams and the 
statistical multiplexing data at a first rate greater than a data transmission 
rate on a transmission channel of a subsequent stage, and outputs first data 
including the statistical multiplexing data. In addition, the multiplexing 
means conducts multiplexing processing on data, obtained by removing the 
statistical multiplexing data from the data outputted from the respective 
encoding means, at a second rate equal to a data transmission rate on the 
transmission channel of the subsequent stage, and outputs second data 
which does not include the statistical multiplexing data to the transmission 
channel of the subsequent stage. The encoding control means acquires the 
statistical multiplexing data of the respective encoding means from the first 
data outputted from the multiplexing means, and conducts control using 
statistical multiplexing on the respective encoding means on the basis of the 
statistical multiplexing data. 

In a video data multiplexing control method which is used in a video 
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multiplexing device including a plurality of encoding means for encoding 
program data respectively including video data and outputting encoded 
streams, multiplexing means for multiplexing the encoded streams 
outputted from the respective encoding means, and encoding control means 
for controlling the respective encoding means, wherein control using 
statistical multiplexing is conducted on the respective encoding means by 
encoding control means, comprises: a statistical multiplexing data output 
procedure in the encoding means for generating statistical multiplexing data 
required for control using statistical multiplexing, and outputting the 
generated data on the same transmission channels as encoded streams are 
transmitted; a multiplexing procedure in the multiplexing means for 
acquiring the encoded streams and the statistical multiplexing data from the 
respective encoding means via the transmission channels, conducting 
multiplexing processing on the encoded streams and the statistical 
multiplexing data at a first rate greater than a data transmission rate on a 
transmission channel of a subsequent stage, outputting first data including 
the statistical multiplexing data, conducting multiplexing processing on data 
obtained by removing the statistical multiplexing data from the data 
outputted from the respective encoding means, at a second rate equal to a 
data transmission rate on the transmission channel of the subsequent stage, 
and outputting second data which does not include the statistical 
multiplexing data to the transmission channel of the subsequent stage; and 
an encoding control procedure in the encoding control means for acquiring 
the statistical multiplexing data of the respective encoding means from the 



- 18 - 

first data outputted from the multiplexing means, and conducting control 
using statistical multiplexing on the respective encoding means on the basis 
of the statistical multiplexing data. 

In another video data multiplexing control method of the present 
invention, statistical multiplexing data required for control using statistical 
multiplexing is generated and outputted on the same transmission channels 
as the encoded streams, by the statistical multiplexing data output 
procedure in the encoding means. Furthermore, by the multiplexing 
procedure in the multiplexing means, the encoded streams and the 
statistical multiplexing data are acquired from the respective encoding 
means via the transmission channels, subjected to multiplexing processing 
at a first rate greater than a data transmission rate on a transmission 
channel of a subsequent stage, and first data including the statistical 
multiplexing data is outputted. In addition, data obtained by removing the 
statistical multiplexing data from the data outputted from the respective 
encoding means are subjected to multiplexing processing at a second rate 
equal to a data transmission rate on the transmission channel of the 
subsequent stage, and second data which does not include the statistical 
multiplexing data is outputted to the transmission channel of the 
subsequent stage. Furthermore, by the encoding control procedure in the 
encoding control means, the statistical multiplexing data of the respective 
encoding means is acquired from the first data outputted from the 
multiplexing means, and control using statistical multiplexing is conducted 
on the respective encoding means on the basis of the statistical multiplexing 
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data. 

An encoded stream multiplexing device for multiplexing encoded 
streams according to the present invention comprises: a plurality of encoding 
means for respectively encoding video data of a plurality of channels on the 
basis of target encoding rates and outputting encoded video streams; 
encoding control means for computing the target encoding rates supplied to 
the plurality of encoding means for respective channels; and multiplexing 
means for multiplexing a plurality of encoded streams respectively outputted 
from the plurality of encoding means. The plurality of encoding means 
output the encoded video streams as video transport stream packets, and 
output encoding difficulty information indicating encoding difficulties in 
encoding video data of the plurality of channels as private transport stream 
packets. The multiplexing means includes a multiplexing circuit for 
receiving a plurality of transport streams including the video transport 
stream packets and the private transport stream packets respectively from 
the plurality of encoding means, multiplexing the plurality of transport 
streams, and thereby generating a multiplexed transport stream. The 
encoding control means receives the multiplexed transport stream from the 
multiplexing means, extracts the private transport stream packet included 
in the multiplexed transport stream, and computes the target encoding rates 
respectively corresponding to the plurality of channels on the basis of the 
encoding difficulty information described in the extracted private transport 
stream packets. 

Another encoded stream multiplexing device for multiplexing 
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encoded streams according to the present invention comprises: a plurality 
of encoding means for respectively encoding video data of a plurality of 
channels on the basis of target encoding rates, thereby generating encoded 
video streams, and outputting the encoded video streams as video transport 
stream packets, outputting encoding difficulty information indicating 
encoding difficulties in encoding video data of the plurality of channels as 
private transport stream packets; multiplexing means for receiving a 
plurality of transport streams including the video transport stream packets 
and the private transport stream packets respectively from the plurality of 
encoding means, multiplexing the plurality of transport streams, and 
thereby generating a multiplexed transport stream; and encoding control 
means for receiving the multiplexed transport stream from the multiplexing 
means, extracting the private transport stream packet included in the 
multiplexed transport stream, computing the target encoding rates 
respectively corresponding to the plurality of channels on the basis of the 
encoding difficulty information described in the extracted private transport 
stream packets, supplying the computed target encoding rates respectively 
to the plurality of encoding means, and thereby controlling rates of the 
encoded streams outputted from the plurality of encoding means. 

An encoded stream multiplexing method for multiplexing a 
plurality of encoded streams generated by encoding video data of a plurality 
of channels according to the present invention comprises the steps of: 
encoding video data of the plurality of channels, generating a plurality of 
encoded streams, and calculating encoding difficulty information indicating 
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encoding difficulties in encoding video data of the plurality of channels; 
outputting the plurality of encoded streams as video transport stream 
packets, and outputting the encoding difficulty information as private 
transport stream packets; respectively receiving a plurality of transport 
streams including the video transport stream packets and the private 
transport stream packets, multiplexing the plurality of transport streams, 
and thereby generating a multiplexed transport stream; and receiving the 
multiplexed transport stream, extracting the private transport stream 
packets included in the multiplexed transport stream, and computing the 
target encoding rates respectively corresponding to the plurality of channels 
on the basis of the encoding difficulty information described in the extracted 
private transport stream packets. 

Another encoded stream multiplexing method for multiplexing a 
plurality of encoded streams according to the present invention comprises: a 
plurality of encoding steps for respectively encoding video data of a plurality 
of channels on the basis of target encoding rates, thereby generating encoded 
video streams, outputting the encoded video streams as video transport 
stream packets, and outputting encoding difficulty information indicating 
encoding difficulties in encoding video data of the plurality of channels as 
private transport stream packets; a multiplexing step for receiving a 
plurality of transport streams including the video transport stream packets 
and the private transport stream packets respectively from the plurality of 
encoding steps, multiplexing the plurality of transport streams, and thereby 
generating a multiplexed transport stream; and an encoding control step for 
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receiving the multiplexed transport stream from the multiplexing step, 
extracting the private transport stream packets included in the multiplexed 
transport stream, computing the target encoding rates respectively 
corresponding to the plurality of channels on the basis of the encoding 
difficulty information described in the extracted private transport stream 
packets, supplying the computed target encoding rates respectively to the 
plurality of encoding steps, and thereby controlling rates of the encoded 
streams outputted from the plurality of encoding steps. 

In the encoded stream multiplexing device or the encoded stream 
multiplexing method according to the present invention, a plurality of 
transport streams including the video transport stream packets and the 
private transport stream packets, which include the encoding difficulty 
information indicating the encoding difficulties at the time of encoding video 
data of the plurality of channels, are multiplexed. A multiplexed transport 
stream is thus generated. A private transport stream packets included in 
the multiplexed transport stream are extracted. On the basis of the 
encoding difficulty information described in the extracted private transport 
stream packets, the target encoding rates respectively corresponding to the 
plurality of channels are computed. 

An encoding device for encoding video data of a plurality of channels 
according to the present invention comprises: a plurality of encoding means 
for respectively outputting a plurality of encoded video streams generated by 
encoding the video data of the plurality of channels as video transport 
stream packets, and outputting encoding difficulty information indicating 
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encoding difficulties in encoding video data of the plurality of channels as 
private transport stream packets; and encoding control means for computing 
target encoding rates respectively corresponding to the plurality of channels 
on the basis of the encoding difficulty information described in the private 
transport stream packets outputted from the plurality of encoding means, 
supplying the computed target encoding rates respectively to the plurality of 
encoding means, and thereby controlling rates of the encoded streams 
outputted from the plurality of encoding means. 

An encoding method for encoding video data of a plurality of 
channels according to the present invention comprises: outputting a 
plurality of encoded video streams generated by encoding the video data of 
the plurality of channels by using a plurality of encoding means as video 
transport stream packets, and outputting encoding difficulty information 
indicating encoding difficulties in encoding video data of the plurality of 
channels as private transport stream packets; and computing the target 
encoding rates respectively corresponding to the plurality of channels on the 
basis of the encoding difficulty information described in the outputted 
private transport stream packets, supplying the computed target encoding 
rates respectively to the plurality of encoding means, and thereby controlling 
rates of the encoded streams outputted from the plurality of encoding means. 

In the encoding device or the encoding method according to the 
present invention, a plurality of encoded video streams generated by 
encoding the video data of the plurality of channels are outputted as video 
transport stream packets. In addition, encoding difficulty information 
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indicating encoding difficulties in encoding video data of the plurality of 
channels is outputted as private transport stream packets. On the basis of 
the encoding difficulty information described in the outputted private 
transport stream packets, the target encoding rates respectively 
corresponding to the plurality of channels are computed, and thereby rates of 
the encoded streams are controlled. 

Other objects, features, and advantages of the present invention will 
become apparent by the following description. 

Brief Description of Drawings 

FIG. 1 is a block diagram showing a schematic configuration of a 
digital broadcasting system of a related art. 

FIG. 2 is a block diagram showing a schematic configuration of an 
image encoding device illustrated in FIG. 1. 

FIG. 3 is a block diagram showing a schematic configuration of a 
multiplexing device using statistical multiplexing of a related art. 

FIG. 4 is a block diagram showing a schematic configuration of an 
image encoding device illustrated in FIG. 3. 

FIGS. 5A, 5B and 5C are diagrams illustrating bit rate changes in 
respective image encoding devices of FIG. 

FIG. 6 is a block diagram showing a schematic configuration of a 
multiplexing device using a general purpose computer instead of a statistical 
multiplexing controller illustrated in FIG. 3. 

FIG. 7 is a block diagram showing a schematic configuration of a 
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statistical multiplexing system serving as an image data multiplexing device 
according to an embodiment of the present invention. 

FIG. 8 is a block diagram showing a configuration of an encoding 
device illustrated in FIG. 7. 

FIG. 9 is a block diagram showing a configuration of a video encoder 
illustrated in FIG. 8. 

FIG. 10 is a block diagram showing a configuration of a multiplexer 
in an embodiment of the present invention. 

FIG. 11 is a block diagram showing a configuration of a 
multiplexing control circuit illustrated in FIG. 10. 

FIG. 12 is a diagram showing a configuration of a TS packet. 

FIG. 13 is a diagram for description of contents of a private packet 
in an embodiment of the present invention. 

FIG. 14 is a diagram for description of contents of a private packet 
in an embodiment of the present invention. 

FIG. 15 is a diagram showing a transport stream in case that 
private packets are simply removed from all TS packets. 

FIG. 16 is a diagram showing a transport stream containing the 
private packet. 

FIG. 17 is a diagram for description of operation timing of the 
multiplexing control circuits illustrated in FIG. 11. 

FIG. 18 is a diagram for description of operation timing of the 
multiplexing control circuit illustrated in FIG. 11. 

FIG. 19 is a flow chart showing operation relating to statistical 
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multiplexing of encoding devices in an embodiment of the present invention. 

FIG. 20 is a flow chart showing an example of operation of a 
statistical multiplexing computer in an embodiment of the present invention. 

FIG. 21 is a characteristic diagram for describing the operation 
illustrated in FIG. 20. 

FIG. 22 is a characteristic diagram for describing the operation 
illustrated in FIG. 20. 

FIG. 23 is a flow chart showing provisional bit rate correction 
processing illustrated in FIG. 20. 

FIG. 24 is a characteristic diagram for describing the operation 
illustrated in FIG. 20. 

FIG. 25 is a characteristic diagram for describing the operation 
illustrated in FIG. 20. 

Best Mode for Carrying Out the Invention 

Hereinafter, embodiments of the present invention will be described 
in detail by referring to the drawings. 

FIG. 7 is a block diagram showing a configuration of a statistical 
multiplexing system serving as an image (video) data multiplexing device 
according to an embodiment of the present invention. This statistical 
multiplexing system 1 using a MPEG system, includes: a plurality of 
encoding devices 2 l to 2 n as an encoding means inputting video data V, to V n 
(where n is an integer value of at the minimum 2) and audio data A x to 
serving as program data in the present invention, conducting compression 
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encoding on the video data and the audio data, and outputting transport 
streams TS X to TS n , which are encoded data sequences (encoded streams) in 
the MPEG system; a statistical multiplexing computer 3 as an encoding 
control means, connected to the respective encoding devices 2 i (where i is an 
arbitrary integer value ranging 1 to n) via a network, such as an Ethernet 5, 
and conducting bit rate control of feedforward type; the multiplexer 4 as a 
multiplexing means, respectively inputting transport streams TS i; 
respectively outputted from the encoding devices 2 4 via transmission 
channels 6^ multiplexing the transport streams TS i; and outputting 
multiplexed transport streams c and TS m to the statistical multiplexing 
computer 3 and such as modulator not shown in figures, respectively. In 
each encoding device 2 i5 a port for the Ethernet 5 is provided. Furthermore, 
a general purpose computer can be used as the statistical multiplexing 
computer 3. 

The multiplexer 4 acquires encoded data (encoded streams) and 
data for statistical multiplexing from the respective encoding devices 2 i via 
the transmission channels 6^ conducts multiplexing processing on them at a 
first speed (hereinafter referred to as rate) Rl, larger than a data 
transmission speed (hereinafter referred to as transmission rate), on a 
transmission channel of a subsequent stage, and outputs the transport 
stream TS d serving as first data containing data for statistical multiplexing. 
In addition, the multiplexer 4 conducts multiplexing processing on data 
obtained by removing the data for statistical multiplexing from the data 
outputted from the respective encoding devices 2 V at a second rate R2 which 
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is equal to the transmission rate on the transmission channel of the 
subsequent stage, and outputs the transport stream TS m serving as second 
data which does not contain the data for statistical multiplexing to the 
transmission channel of the subsequent stage. 

Each encoding device 2 i encodes inputted video data V ; and audio 
data A; while deriving an encoding difficulties D> concerning an image to be 
encoded, as data for statistical multiplexing by using the video data V ; . 
Each encoding device 2j converts the encoded video data to a video packet 51, 
for example, by one packet per one frame, converts the encoded audio data to 
an audio packet 52, converts the encoding difficulty value Dj to a private 
packet 53, and outputs them to the multiplexer 4 as the transport stream TSj. 
In addition, the packet forming the transport stream TS i is called transport 
stream packet (hereinafter referred to as TS packet). 

The multiplexer 4 multiplexes all packets including the video 
packets, audio packets, and private packets of the transport streams TSi to 
TS n supplied from n encoding devices 2 1 to 2 n , and outputs the multiplexed 
packets to the statistical multiplexing computer 3 as the transport stream 
TS d . In addition, the multiplexer 4 outputs the transport stream TS m , 
obtained by removing the private packets from the multiplexed transport 
stream TS d , to the modulator and the like via the transmission channel of 
the subsequent stage. 

The statistical multiplexing computer 3 takes out private packets 
from the transport stream TS d sent from the multiplexer 4, derives target bit 
rates Rate 4 for the respective encoding devices 2 ; on the basis of the encoding 
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difficulties obtained from the private packets, and returns target bit rate 
data representing the target bit rates Rate; to the respective encoding 
devices 2 ; . In addition, taking out the private packets can be implemented 
by hardware, such as an internal board, or software. 

On the basis of the target bit rate Rate £ thus set, each encoding 
device 2 { conducts bit rate control and conducts compression encoding on the 
video data Vj. 

FIG. 8 is a block diagram showing a detailed internal configuration 
of the encoding device % illustrated in FIG. 7. As shown in FIG. 8, the 
encoding device 2 i includes: a video encoder 10 for conducting compression 
encoding on the video data V £ and outputting a video stream VS{, an FIFO 
(first-in first-out) memory 62a for outputting the video stream VS; supplied 
from the video encoder 10 after delaying the video stream for a 
predetermined time and outputting the delayed video stream, an audio 
encoder 60 for conducting compression encoding on the audio data A> and 
outputting an audio stream AS^ an FIFO (first-in first-out) memory 62b for 
outputting the audio stream AS { supplied from the audio encoder 60 delaying 
for a predetermined time; a switch 64 having one fixed contact 64a connected 
to the FIFO memory 62a and the other fixed contact 64b connected to the 
FIFO memory 62b and selectively outputting either the video stream VS i or 
the audio stream AS; from a movable contact 64c thereof; a switch 68 having 
one fixed contact 68a connected to the movable contact 64c of the switch 64; 
and a FIFO memory 69 for delaying output data from a movable contact 68c 
of the switch 68 for a predetermined time and outputting the delayed data as 
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the transport stream TSj. 

The encoding device 2 i further includes a CPU (central processing 
unit) 65, a RAM (random access memory) 66 serving as a working area, and 
a ROM (read only memory) 67, which are connected to each other via a CPU 
bus 71. The other fixed contact 68b of the switch 68 is connected to the 
RAM 66. 

The encoding device 2j further includes an interface 61 for 
outputting bits quantity generated per picture frame by the compression 
encoding in the video encoder 10 onto the CPU bus 71, an interface 63 for 
outputting data of a quantity generated by the compression encoding in the 
audio encoder 60 onto the CPU bus 71, an Ethernet interface 70 for inputting 
private packets transmitted from the statistical multiplexing computer 3 to 
the image encoding device 2 i via the Ethernet 5, and an interface 72 for ' 
setting the target bit rate Rate t in the video encoder 10. The interfaces 61, 
63 and 72, and the Ethernet interface 70 are connected to the CPU bus 71, 
respectively. 

On the basis of a switchover instruction signal S x supplied from the 
CPU 65 via the CPU bus 71, the switch 64 conducts switchover between the 
video stream VSi supplied from the FIFO memory 62a and the audio stream 
AS ; supplied from the FIFO memory 62b to output the result to the switch 68. 
On the basis of a switchover instruction signal S 2 supplied from the CPU 65 
via the CPU bus 71, the switch 68 conducts switchover between output data 
of the switch 64 and data, such as the encoding difficulties D i5 supplied from 
the RAM 66 to output the result to the FIFO memory 69. 
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FIG. 9 is a block diagram showing a detailed configuration of the 
video encoder 10 illustrated in FIG. 8. As shown in FIG. 9, the video 
encoder 10 includes: an encoder control unit 11 for receiving the video data Vj 
and conducting preprocessing and the like for compression encoding; an 
FIFO memory 12 for outputting output data of the encoder control unit 11 
delaying for a predetermined time; an encoding unit 13 for receiving output 
data of the FIFO memory 12, conducting compression encoding by an 
encoding method according to a picture type of each picture, and outputting 
the video stream VSj which is an encoded data sequence; a motion detection 
circuit 14 for detecting a motion vector on the basis of the output data of the 
encoder control unit 11 and sending the result to the encoding unit 13; and 
an encoding control unit 15 for controlling the encoding unit 13 on the basis 
of intra AC data Sa t outputted from the encoder control unit 11 and ME 
residual data Sz ; outputted from the motion detection circuit 14. 

The encoder control unit 11 includes: an image rearrangement 
circuit 21 for receiving the video data Vj and rearranging the order of 
pictures (I picture, P picture, and B picture) according to the order of 
encoding, a scan conversion/macro blocking circuit 22 for receiving output 
data of the image rearrangement circuit 21, determining whether the output 
data has a frame structure or a field structure, conducting scan conversion 
and macro blocking of 16 X 16 pixels according to the result of the 
determination; and an intra AC computation circuit 23 for receiving output 
data of the scan conversion/macro blocking circuit 22, calculating intra AC in 
the I picture, sending intra AC data Sai to the encoding control unit 15, while 
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sending the output data of the scan conversion/macro blocking circuit 22 to 
the FIFO memory 12 and the motion detection circuit 14. In addition, the 
intra AC is defined in the I picture as sum total of absolute values of 
difference values between pixel values of respective pixels in an 8 X 8 pixel 
DCT (discrete cosine transform) block and a mean value of pixel values in 
the DCT block. It can be said that the intra AC represents the degree of 
complication. 

The encoding unit 13 includes: a subtraction circuit 31 deriving a 
difference between the output data of the FIFO memory 12 and predicted 
image data; a DCT circuit 32 conducting DCT on output data of the 
subtraction circuit 31, taking a DCT block as a unit and outputting a DCT 
coefficient; a quantization circuit 33 quantizing output data of the DCT 
circuit 32; a variable length encoding circuit 34 conducting variable length 
encoding on output data of the quantization circuit 33; a buffer memory 35 
temporarily holding output data of the variable length encoding circuit 34 
and outputting it as the video stream VS £ ; an inverse quantization circuit 36 
conducting inverse quantization on the output data of the quantization 
circuit 33; an inverse DCT circuit 37 conducting inverse DCT on output data 
of the inverse quantization circuit 36; an addition circuit 38 adding output 
data of the inverse DCT circuit 37 to the predicted image data and 
outputting the resultant sum; and a motion compensation circuit 39 for 
holding output data of the addition circuit 38, conducting motion 
compensation according to the motion vector sent from the motion detection 
circuit 14, and outputting predicted image data to the subtraction circuit 31 
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and the addition circuit 38. 

On the basis of the output data of the encoder control unit 11, the 
motion detection circuit 14 searches for a target macro block of a picture to 
be subjected to compression encoding, and a macro block, that the sum of 
absolute values or the sum of squared values of difference values of pixel 
value between the macro block and the noticeable macro block, becomes 
minimum in the referred picture, and detects the motion vector and sends it 
to the motion compensation circuit 39. Furthermore, when deriving the 
motion vector, the motion detection circuit 14 sends the sum of absolute 
values or the sum of squared values of difference values in pixel value, 
between the macro block providing the minimum value and the target macro 
block, to the encoding control unit 15 as the ME residual data Szj. 

The encoding control unit 15 includes a ME residual calculation unit 
41 for calculating an ME residual which is the value of adding the ME 
residual data Sz; together supplied from the motion detection circuit 14 over 
the whole picture, and an encoding difficulty calculation unit 42 for 
calculating an encoding difficulty value D s representing the encoding 
difficulty of the picture, on the basis of the ME residual calculated by the ME 
residual calculation unit 41 and the intra AC data supplied from the 
intra AC computation circuit 23, and outputting the encoding difficulty value 
D, to the interface 61. 

The encoding control unit 15 further includes a quantization index 
determining unit 45 for determining a quantization index corresponding to a 
quantization characteristic value in the quantization circuit 33 so as to make 
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the bit rate equal to the target bit rate Rate s extracted from the target bit 
rate data which is sent from the statistical multiplexing computer 3, and 
sending the determined quantization index to the quantization circuit 33. 

FIG. 10 is a block diagram showing a detailed internal configuration 
of the multiplexer 4. As shown in FIG. 10, the multiplexer 4 includes: a 
multiplexing unit 101 acquiring the transport streams TS { from the 
respective encoding devices 2 4 via the transmission channels 6^ conducting 
multiplexing processing on them, and outputting the transport stream TS d 
including private packets; and a private packet removing unit 102 removing 
the private packets from data multiplexed by the multiplexing unit 101 and 
outputting resultant data, to the modulator and the like via the transmission 
channel of the subsequent stage, as the transport stream TS m . 

The private packet removing unit 102 includes a multiplexing 
control circuit 103, serving as a first multiplexing control unit, for controlling 
the multiplexing unit 101 so as to output the transport stream TS d from the 
multiplexing unit 101 at the first rate Rl and holding data obtained by 
removing private packets from the transport stream TS d outputted from the 
multiplexing unit 101; and a multiplexing control circuit 104 for controlling 
the multiplexing control circuit 103 so as to output the data held by the 
multiplexing control circuit 103 to the transmission channel of the 
subsequent stage at the second rate R2, as the transport stream TS m . 

FIG. 11 is a block diagram showing detailed configurations of the 
multiplexing control circuits 103 and 104 illustrated in FIG. 10. As shown 
in FIG. 11, the multiplexing control circuit 103 includes: a private packet 
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PID table 111 holding packet identification information (hereinafter referred 
to also as PID) (i) of private packets of each encoding device 2 { ; a rate control 
circuit 112 for sending a request signal REQ1 requesting the multiplexing 
unit 101 to send packets; a PID extraction circuit 113 extracting a PID from 
a packet outputted from the multiplexing unit 101 and outputting the PID; a 
PID comparison circuit 114 comparing the PID, outputted from the PID 
extraction circuit, with the PID (i) registered in the private packet PID table 
111; a delay circuit 115 receiving a packet outputted from the multiplexing 
unit 101, outputting the packet, delaying for a time period until a 
comparison result is obtained in the PID comparison circuit 114; a memory 
circuit 116 temporarily holding the packet outputted from the delay circuit 
115 for the rate conversion; and a memory control circuit 117 controlling the 
memory circuit 116. 

The PID comparison circuit 114 sends a noncoincidence signal 
"mismatch" to the memory control circuit 117 when the PID outputted from 
the PID extraction circuit 113 does not coincide with the PID (i) registered in 
the packet PID table 111. In the case of coincidence, the PID comparison 
circuit 114 does not send the noncoincidence signal "mismatch" to the 
memory control circuit 117. Upon receiving the noncoincidence signal 
"mismatch," the memory control circuit 117 sends a write signal "write" to 
the memory circuit 116. Upon receiving the write signal "write," the 
memory circuit 116 writes the packet outputted from the delay circuit 115 
therein. Through this operation, packets other than the private packet 53 
are written into the memory circuit 116. 
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The multiplexing control circuit 104 includes a rate control circuit 
118 for sending a request signal REQ2 requesting the memory control circuit 
117 to send packets. Upon receiving the request signal REQ2, the memory 
control circuit 117 sends a read signal "read" to the memory circuit 116. 
Each time when the memory circuit 116 receives the read signal "read," the 
memory circuit 116 reads the packets held therein one by one in written 
order, and outputs the packets as the transport stream TS m . This transport 
stream TS m passes through the multiplexing control circuit 104 and is 
outputted from the private packet removing unit 102. 

The rate control circuit 112 sends the request signal REQ1 to the 
multiplexer 4 so as to make the transport stream TS d be outputted from the 
multiplexing unit 101 at a rate Rl which is larger than the sum of a 
transmission rate Rt on the transmission channel of the stage subsequent to 
the multiplexer 4 and a rate Rp of the private packet. The relationship of 
the rates Rt, Rp and Rl is shown by the following expression (3). 

Rl ^ Rt + Rp -(3) 

On the other hand, the rate control circuit 118 sends the request 
signal REQ2 to the memory control circuit 117 so as to make the transport 
stream TS m be outputted from the memory circuit 116 at a rate equal to the 
transmission rate Rt. The relationship of the rates Rl and R2 is shown by 
the following expression (4). 



R1>R2 



...(4) 



Furthermore, the memory control circuit 117 monitors the number 
of packets entering and exiting the memory circuit 116 so as to send a wait 
signal "wait" to the rate control circuit 112 when the memory circuit is likely 
to overflow, the memory control circuit 117. Upon receiving the wait signal 
"wait," the rate control circuit 112 does not issue the request signal REQ1 
even if it's a timing of issuing the request signal REQ1. 

By referring to FIG. 12, a configuration of a TS packet will be 
described briefly. The TS packet is formed of a transport header section 
(hereinafter referred to as TS header section) of 4 bytes and a payload 
section in which actual data of 184 bytes are recorded. As a whole, the TS 
packet has 188 bytes. The TS header section is formed of: a synchronizing 
byte having 8 unique bits for indicating the start of the TS packet; an error 
indicator section for indicating whether a bit error is present or not in the TS 
packet; a unit start indication section for indicating whether a head of a 
packetized elementary stream (PES) packet is present in this TS packet; a 
transport packet priority section for indicating the degree of importance of 
this TS packet; a PID section for storing a PID indicating the kind of stream 
data accommodated in the payload section of this TS packet; a scramble 
control section for indicating whether the stream data accommodated in the 
payload section has been subjected to scrambling or not; an adaptation field 
control section for indicating whether an adaptation field section and the 
payload section are present or not; a cyclic counter section for storing cyclic 
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counter information used to detect whether a TS packet having the same 
PID has been rejected on the way or not; and an adaptation field section for 
storing various kinds of control information. 

The adaptation field section is formed of: an adaptation field length 
indicating the length of the adaptation field section; a discontinuity 
indicating section for indicating whether time information has been reset in 
a TS packet of the same stream subsequent to this TS packet; a random 
access indicating section for indicating whether this TS packet is an entry 
point of random access or not; a stream preference indicating section 
indicating whether an important part of the stream data has been stored in 
the pay load section of this TS packet or not; a flag control section for storing 
flag information concerning a conditional encoding section; a conditional 
encoding section for storing reference time information called PCR (Program 
Clock Reference), reference time information called OPCR (Original Program 
Clock Reference), or information such as splice count down indicating an 
index as far as a data replacement point, and a stuffing byte section for 
stuffing invalid data bytes in order to adjust the data length. 

By referring to FIGs. 13 and 14, contents of a private packet in the 
present embodiment will now be described. FIGs. 13 and 14 represent data 
formed as packets (TS packets) on a transmission channel. In case that the 
transmission channel ^ is, for example a serial transmission channel of 270 
Mbit/second called DVB-Serial-ASI (Asynchronous Serial Interface) defined 
in DVB-TM (exactly speaking, Interfaces for CATV/SMATV Headends and 
similar Professional Equipment issued by DVB-TM Ad hoc Group Physical 
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Interface), data are carried intermittently as shown in FIGs. 13 and 14. In 
the examples shown in FIGs. 13 and 14, a video packet 51, an audio packet 
52, and a private packet 53 are illustrated as intermittent data. In each of 
these packets, the head portion of which is a TS header section as described 
above, and the other portion than the TS header section is a payload section. 
In the payload section, actual data is described. 

In the TS header section, a PID is described. The PID is an 
identification (ID) number of a packet defined in the MPEG standards in 
order to identify the attribute of the packetized data. The PID must be set 
for each of video channels and each of audio channels. In the present 
embodiment, it is necessary to secure an individual PID for the private 
packet as well. In FIGs. 13 and 14, a PID of the video packet 51 is 
represented as PID-V, a PID of the audio packet 52 is represented as PID-A, 
and a PID of the private packet 53 is represented as PID-P. 

In case that the encoding difficulties of each encoding device % x is 
transmitted by using the private packet, it is necessary to make it possible to 
determine from which encoding device 2, the encoding difficulties is supplied. 
As the method for that purpose, the following two methods are conceivable. 

The first method is a method to use a PID section in a TS header 
section is used. FIG. 13 shows simplified contents of the private packet 53. 
As shown in FIG. 13, the private packet 53 includes a PID section 53a in its 
TS header section, a cyclic counter section 53b, an adaptation field section 
53c, and a payload section 53d. In the payload section 53d, an encoding 
difficulties 53e is included. In the first method, as many PIDs as the 
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encoding devices 2 4 are secured, and one PID is set for each encoding device 2 { 
in the PID section 53a shown in FIG. 13. In this case, the statistical 
multiplexing computer 3 receives encoding difficulties respectively for 
specified PIDs, and calculates target bit rates Ratej respectively for the 
encoding devices 2 V Furthermore, in this method, packet rejection is 
detected by using cyclic counter information stored in the cyclic counter 
section 53b shown in FIG. 13. Since the cyclic counter information is 
provided in a transport layer in this method, it becomes possible to detect the 
packet rejection on the transport layer level. Furthermore, in this method, 
it is necessary to secure as many PIDs as the encoding devices 2 V and in 
addition, it is necessary to conduct removal of private packets from the 
transport stream TS ; by using as many PIDs as the encoding devices 2 t . 

The second method is a method securing one PID with regard to all 
private packets determining an encoding device identification number for 
each encoding device 2 i separately from the PID, and identifying from which 
encoding device 2 i the encoding difficulty is supplied by using the 
identification number. In FIG. 14, contents of the payload section 53d of 
the private packet 53 when the second method is used are shown. In this 
case, the payload section 53d includes: an encoding device identification 
number 53f for identifying from which encoding device 2 X the encoding 
difficulties is supplied; a cyclic counter 53g provided besides the cyclic 
counter in the transport layer, in order to detect whether packet rejection 
has been conducted or not; and encoding difficulties 53e supplied from the 
respective encoding devices 2 ; . 
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By the way, the private packet, containing the encoding difficulty 
values, is not transmitted to the receiving side as it is data which is not used 
on the receiving side, but used only on the sending side. FIG.15 shows an 
example of the transport stream transmitted to the receiving side when the 
private packets containing the encoding difficulty values are simply removed 
from all TS packets in the multiplexer 4. In this example, a video packet 
51i supplied from the encoding device 2 X , and a video packet 51 2 supplied 
from the encoding device 2 2 are successively transmitted. Then, a space 
corresponding to one packet of the removed private packet is transmitted. 
Then, an audio packet 52 3 supplied from the encoding device 2 3 , a video 
packet 51 3 supplied from the encoding device 2 3 , and a video packet 51 4 
supplied from the encoding device 2 4 are transmitted sequentially. 

On the other hand, FIG. 16 shows an example of a transport stream 
containing the private packets sent to the statistical multiplexing computer 
3. In this example, a video packet 51 x supplied from the encoding device 2 U 
and a video packet 51 2 supplied from the encoding device 2 2 are successively 
transmitted. Then, a private packet 53 is transmitted. Then, an audio 
packet 52 3 supplied from the encoding device 2 3 , a video packet 51 3 supplied 
from the encoding device 2 3 , and a video packet 51 4 supplied from the 
encoding device 2 4 are transmitted one after another. 

In the case of the transport stream as shown in FIG. 15, a time slot 
in which the private packet existed becomes a blank. Therefore, the 
transmission efficiency drops, resulting in the occurrence of an overhead of 
the private packets, i.e. an extra time used to control the private packets 
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occurs. 

Here, the ratio of the private packets for conducting control using 
the statistical multiplexing occupying the transmission rate will now be 
considered. If one frame (1/30 second) is taken as a unit for control, data 
(encoding difficulty) for statistical multiplexing corresponding to one packet 
is generated at intervals of that time. For example, assuming that one 
packet is 188 bytes, one encoding device 2 i occupies the rate of 

188 X 30 = 0.045 (Mbps). 
When packets are transmitted from 10 encoding devices to one multiplexer 4, 
the rate of the private packets for conducting control using the statistical 
multiplexing increases to 10 times, becoming 0.45 Mbps. Since the 
transmission rate has a band of approximately 30 Mbps in the ordinary 
digital CS broadcasting or the like, an overhead of the private packets 
equivalent to approximately 1.5% occurs. Furthermore, when more 
complicated control is attempted, the number of packets increases to several 
times and the overhead of the private packets is expected to occupy several 
percents of the band. Originally, utilization of 100% of the band is ideal, 
but as a matter of fact, several percents of unutilized part of band exist. 
Moreover when the rate of several percents is occupied by the overhead of 
the private packets, the efficiency of the transmission rate is further 
aggravated. 

In the present embodiment, therefore, data supplied from the 
respective encoding devices 2 { are multiplexed by the multiplexer 4 at a first 
rate Rl larger than the transmission rate on the transmission channel of the 
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subsequent stage to output the transport stream TS d containing the private 
packets for statistical multiplexing to the statistical multiplexing computer 3. 
In addition, data obtained by removing the private packets for statistical 
multiplexing from the data outputted from the respective encoding devices 2; 
are multiplexed by the multiplexer 4 at a second rate R2 equivalent to the 
transmission rate on the transmission channel of the subsequent stage to 
output the transport stream TS m not containing the private packets for 
statistical multiplexing to the transmission channel of the subsequent stage. 

Operation of the statistical multiplexing system 1 shown in FIG. 7 
will now be described. The following description serves also as description 
of the image data multiplexing control method according to the present 
embodiment. In this statistical multiplexing system 1, the video data Vj 
and the audio data Aj are encoded by each encoding device 2j. For each 
video data V i5 each encoding device 2 { forms the private packet 53 containing 
the encoding difficulties B i which represents the encoding difficulty 
concerning the image to be encoded. Along with the video packet 51 of 
encoded video data and the audio packet 52 of encoded audio data, each 
encoding device 2 £ outputs the private packet 53 to the multiplexer 4 via the 
transmission channel 6j as the transport stream TS t . 

Subsequently, as shown in FIG. 10, the multiplexer 4 multiplexes 
the transport streams TS X to TS n supplied from the encoding devices 2 { in the 
multiplexing unit 101, and generates the multiplexed transport stream TS d . 
The transport stream TS d includes the private packet 53 containing the 
encoding difficulties D x to D n therein, and is sent to the statistical 
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multiplexing computer 3 in order to conduct the control using the statistical 
multiplexing. Furthermore, the transport stream TS d is inputted to the 
private packet removing unit 102 as well. From this private packet 
removing unit 102, the transport stream TS m obtained by removing the 
private packet 53 from the transport stream TS d is outputted to the 
modulator and the like via the transmission channel of the subsequent stage. 

The statistical multiplexing computer 3 only takes out the private 
packet 53 having the encoding difficulties D L to D n in a packet form from the 
inputted transport stream TS d . The statistical multiplexing computer 3 
determines the target bit rate Rate t for each video data V t by using a 
statistical multiplexing technique on the basis of the encoding difficulty 
described in the private packet thus taken out. The statistical multiplexing 
computer 3 derives the target bit rates Rate! to Rate n for all encoding devices 
2 l to 2 n , and sends them to all encoding devices 2 X to 2 n via the Ethernet 5 as 
target bit rate data. From the target bit rate data, each encoding device 2 { 
extracts the target bit rate Rate t for the encoding device 2 £ . On the basis of 
the target bit rate Rate;, each encoding device 2 i conducts compression 
encoding on the video data Vj, and outputs the transport stream TS t to the 
multiplexer 4. The multiplexer 4 multiplexes the inputted transport 
streams TS i; and outputs the transport streams TS m and TS d . 

Operation conducted by the encoding device 2 £ shown in FIG. 8 in 
this case will now be described in detail. First of all, the video stream VS ; 
supplied from the video encoder 10 is outputted to the FIFO memory 62a. 
Along with the amount of bits generated by compression encoding, the 
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encoding difficulty value Dj is outputted to the interface 61. Along with the 
amount of generated bits, the encoding difficulty value Dj is written into the 
RAM 66 from the interface 61 via the CPU bus 71 by the CPU 65. The 
audio stream AS; supplied from the audio encoder 60 is outputted to the 
FIFO memory 62b. 

By the switchover signal S 2 , the CPU 65 controls the switch 64 and 
thereby selectively outputs one of the video stream VS; and the audio stream 
AS;. Furthermore, by using the switchover signal S lt the CPU 65 controls 
the switch 68 and thereby selectively outputs one of the output data of the 
switch 64 and the data supplied from the RAM 66 to the FIFO memory 69. 
By the way, the data supplied from the RAM 66 includes the data described 
in the packet header and the encoding difficulty D ; . 

Thus, a packet header is added to the video stream VS* to generate a 
video packet, a packet header is added to the audio stream ASj to generate an 
audio packet, and a packet header is added to the encoding difficulty D> to 
generate a private packet at the switch 68. Each packet thus generated is 
outputted to the transmission channel 6 t via the FIFO memory 69 and is 
transmitted to the multiplexer 4 as the transport stream TSj. By the way, 
information such as the PID, adaptation field, and the cyclic counter 
required for packet forming are outputted from the RAM 66 to the switch 68. 

Furthermore, to the Ethernet interface 70, the target bit rate data 
derived in the multiplexing computer 3 are transmitted via the Ethernet 5. 
The CPU 65 temporarily writes the target bit rate data into the RAM 66 via 
the CPU bus 71, extracts a pertinent target bit rate Rate ; , and sends it to the 
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quantization index determining unit 45 of the video encoder 10 via the 
interface 72. On the basis of the target bit rate Rate i; the video encoder 10 
conducts compression encoding on the video data Vj. 

Operation of the video encoder 10 shown in FIG. 9 will now be 
described. First of all, the video data V; is inputted to the encoder control 
unit 11 of the video encoder 10. In the encoder control unit 11, the order of 
pictures (I picture, P picture, and B picture) is rearranged according to the 
order of being encoded by the image rearrangement circuit 21 and 
discrimination whether the video data has a frame structure or a field 
structure is performed by the scan conversion/macro blocking circuit 22, and 
conducts scan conversion and macro blocking according to a result of the 
discrimination. In the case of the I picture, according to the intra AC 
computation circuit 23, intra AC is calculated and the intra AC data Sa^ is 
sent to the encoding difficulty calculation unit 42 of the encoding control unit 
15. Furthermore, the output data of the scan conversion/macro blocking 
circuit 22 is sent to the FIFO memory 12 and the motion detection circuit 14 
via the intra AC computation circuit 23. 

The FIFO memory 12 delays the inputted image data for a time 
period required for the encoding difficulty calculation unit 42 to calculate the 
encoding difficulties of pictures of N frames subsequent to the picture which 
has been finished the encoding, and then outputs the image data to the 
encoding unit 13. The motion detection circuit 14 detects a motion vector 
and sends it to the motion compensation circuit 39, while sending the ME 
residual data Sz, to the ME residual calculation unit 41 of the encoding 
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control unit 15. On the basis of the ME residual data Sz i3 the ME residual 
calculation unit 41 calculates the ME residual, and outputs it to the encoding 
difficulty calculation unit 42. 

On the basis of the intra AC data Sai and the ME residual, the 
encoding difficulty calculation unit 42 calculates the encoding difficulty 
value D ; and outputs it to the interface 61. 

In the case of the I picture, in the encoding unit 13, the output data 
of the FIFO memory 12 is inputted to the DCT circuit 32 as it is, and 
subjected to the DCT therein without deriving a difference between the I 
picture and predicted image data in the subtraction circuit 31. The DCT 
coefficient is quantized by the quantization circuit 33. Output data of the 
quantization circuit 33 is subjected to variable length encoding. Output 
data of the variable length encoding circuit 34 is temporarily held by the 
buffer memory 35, and outputted as the video stream VSi consists of bit 
stream. The output data of the quantization circuit 33 is subjected to 
inverse quantization in the inverse quantization circuit 36. Output data of 
the inverse quantization circuit 36 is subjected to inverse DCT in the inverse 
DCT circuit 37. And output image data of the inverse DCT circuit 37 is 
inputted to the motion compensation circuit 39 via the addition circuit 38, 
and held therein. 

In the case of a P picture, in the encoding unit 13, the motion 
compensation circuit 39 generates predicted image data on the basis of 
image data corresponding to a past I picture or P picture which is held and 
the motion vector supplied from the motion detection circuit 14, and the 
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predicted image data is outputted to the subtraction circuit 31 and the 
addition circuit 38. Also, a difference between the output data of the FIFO 
memory 12 and the predicted image data supplied from the motion 
compensation circuit 39 is derived by the subtraction circuit 31, and is 
subjected to DCT in the DCT circuit 32. The DCT coefficient is quantized by 
the quantization circuit 33. Output data of the quantization circuit 33 is 
subjected to variable length encoding by the variable length encoding circuit 
34. Output data of the variable length encoding circuit 34 is temporarily 
held and outputted as the video stream VS ; by the buffer memory 35. 
Furthermore, output data of the quantization circuit 33 is subjected to 
inverse quantization in the inverse quantization circuit 36. Output data of 
the inverse quantization circuit 36 is subjected to inverse DCT in the inverse 
DCT circuit 37. Output data of the inverse DCT circuit 37 and the 
predicted image data are added by the addition circuit 38. A resultant sum 
signal is inputted to the motion compensation circuit 39, and held therein. 

In the case of a B picture, in the encoding unit 13 ,the motion 
compensation circuit 39 generates predicted image data on the basis of two 
image data corresponding to the I picture or P picture of the past and future 
which are held and two motion vectors supplied from the motion detection 
circuit 14, and outputs the predicted image data to the subtraction circuit 31 
and the addition circuit 38. Furthermore, a difference between the output 
data of the FIFO memory 12 and the predicted image data supplied from the 
motion compensation circuit 39 is derived by the subtraction circuit 31, and 
is subjected to DCT in the DCT circuit 32. The DCT coefficient is quantized 
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by the quantization circuit 33. Output data of the quantization circuit 33 is 
subjected to variable length encoding by the variable length encoding circuit 
34. Output data of the variable length encoding circuit 34 is temporarily 
held and outputted as the video stream VS { by the buffer memory 35. The B 
picture is not held in the motion compensation circuit 39. 

So as to accomplish the target bit rate Ratej acquired from the 
interface 72 and set, the quantization index determining unit 45 determines 
the quantization index corresponding to the quantization characteristic 
value in the quantization circuit 33, and sends it to the quantization circuit 
33. As a result, control using the statistical multiplexing is conducted. 

Next, operation of the multiplexing control circuits 103 and 104 
illustrated in FIG. 11 will now be described. First of all, as many PIDs of 
the private packets of the number as required are registered in the private 
packet PID table 111 of the multiplexing control circuit 103 beforehand. 
The rate control circuit 112 outputs the request signal REQ1 to the 
multiplexer 4 at timing conformed to a rate Rl larger than the sum of the 
transmission rate Rt and the rate Rp of the private packet 53. The 
multiplexer 4 transmits a packet according to the request signal REQ1. 
The sent packet becomes the transport stream TS d containing the private 
packet 53, and is outputted to the statistical multiplexing computer 3 while 
is also outputted to the PID extraction circuit 113 and the delay circuit 115. 

The PID extraction circuit 113 extracts a PID out of packets, and 
outputs the extracted PID to the PID comparison circuit 114. The PID 
comparison circuit 114 compares the PID outputted from the PID extraction 
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circuit 113 with the PID (i) registered in the private packet PID table 111. 
In the case of noncoincidence, the PID comparison circuit 114 sends a 
noncoincidence signal "mismatch" to the memory control circuit 117 and in 
the case of coincidence, does not send the noncoincidence signal "mismatch" 
to the memory control circuit 117. Upon receiving the noncoincidence 
signal "mismatch," the memory control circuit 117 sends the write signal 
"write" to the memory circuit 116. Upon receiving the write signal "write," 
the memory circuit 116 writes the packet outputted from the delay circuit 
115. As a result of these operations, packets except the private packet 53 
are written into the memory circuit 116. In addition, the delay circuit 115 
delays the packet for a time period until a result of comparison is obtained in 
the PID comparison circuit 114, and outputs the packet. 

The rate control circuit 118 in the multiplexing control circuit 104 
outputs the request signal REQ2 to the memory control circuit 117, at timing 
conformed to a rate R2 equivalent to the transmission rate Rt. Upon 
receiving the request signal REQ2, the memory control circuit 117 sends the 
read signal "read" to the memory circuit 116. Each time the memory circuit 
116 receives the read signal "read," the memory circuit 116 reads out the 
held packets one by one in written order, and outputs them as the transport 
stream TS m . This transport stream TS m passes through the multiplexing 
control circuit 104 and is outputted from the private packet removing unit 
102. This transport stream TS m becomes a transport stream which does not 
contain the private packet 53. Transmission rates of the transport streams 
TS d and TS m become Rl and R2, respectively. 
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Since the transport stream inputted to the multiplexer 4 is 
subjected to the rate control, the memory circuit 116 normally does not 
overflow. However, there is also a possibility of occurrence of such an error 
that the memory circuit 116 overflows, since operation is conducted in such a 
state that the rate Rl is larger than the rate R2 (Rl > R2). Therefore in 
order to avoid the overflow of the memory circuit 116, the following control is 
conducted. 

That is, the memory control circuit 17 monitors the number of 
packets entering and exiting the memory circuit 116 and sends the wait 
signal "wait" to the rate control circuit 112 when the memory circuit is likely 
to overflow. Upon receiving the wait signal "wait," the rate control circuit 
112 does not issue the request signal REQ1 even at timing for issuing the 
request signal REQ1. While the wait signal "wait" is being received, a 
packet is not supplied from the delay circuit 115 to the memory circuit 116, 
and consequently a packet is not written into the memory circuit 116 and in 
the memory circuit 116, only the read operation is conducted. As a result, 
the memory circuit 116 is avoided from overflowing. When the memory 
control circuit 117 determines that the monitored number of packets is not a 
number causing overflow, it stops sending the wait signal "wait." When 
sending of the wait signal "wait" is suspended, the rate control circuit 112 
sends the request signal REQ1 at timing conformed to the rate Rl. 
According to such a control method, the memory circuit 116 operates without 
overflowing no matter how much the rate Rl is greater than the rate R2. 
Next, making reference to FIG. 17, operation timing of the 
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multiplexing control circuits 103 and 104 at the normal time will now be 
described. In FIG. 17, (a) represents the request signal REQ1, (b) packets 
in the transport stream TS d , (c) the noncoincidence signal "mismatch," (d) 
the write signal "write," (e) packets outputted from the delay circuit 115, (f) 
the request signal REQ2, (g) the read signal "read," and (h) packets in the 
transport stream TS m . 

As shown in (a), the rate control circuit 112 sends the request signal 
REQ1 at timing conformed to the rate Rl. In response to this request signal 
REQ1, TS packets (n-1) to (n+8) containing the private packets 53 appear as 
the transport stream TS d , as shown in (b). It is now assumed that TS 
packets (n+2) and (n+5) are assumed to be private packets. PIDs of the TS 
packets (n+2) and (n+5) are registered in the private packet PID table 111 
beforehand. 

When processing time for a packet by the PID extraction circuit 113, 
the PID comparison circuit 114, and the private packet PID table 111 is 
assumed to be Tdly, the delay time of the packet in the delay circuit 115 also 
becomes Tdly. Therefore, as shown in (c), therefore, the noncoincidence 
signal "mismatch" is established Tdly after a packet is inputted to the PID 
extraction circuit 113. Tdly is represented by the following expression (5). 

Tdly = T: PID + T:PID(i) + T: match/mismatch ...(5) 

In the expression (5), T:PID is a time required for extracting the 
PID of the inputted packet in the PID extraction circuit. T:PID(i) is a time 
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required for reading PIDs of the required number in those registered in the 
private packet PID table 111. T: match/mismatch is a time required for 
determining whether the PID is coincident in the PID comparison circuit 114 
or not. 

Upon receiving the noncoincidence signal "mismatch" from the PID 
comparison circuit 114, the memory control circuit 117 generates the write 
signal "write", as shown in (d). When the write signal "write" becomes high, 
the memory circuit 116 writes a packet therein. As shown in (e), the delay 
circuit 115 delays the packet by the time Tdly and then outputs it. In the 
example shown in FIG. 17, the noncoincidence signal "mismatch" is not 
outputted from the PID comparison circuit 114 at the time of the packet 
(n+2) and the packet (n+5) as represented by "match 1" and "match 2" in (c). 
Accordingly, the write signal "write" does not become high. As indicated 
with broken lines in (e), therefore, the packet (n+2) and the packet (n+5) are 
not written into the memory circuit 116. 

On the other hand, as shown in (f), the rate control circuit of the 
multiplexing control circuit 104 sends out the request signal REQ2 at timing 
conformed to the rate R2. In response to this request signal REQ2, the 
memory control circuit 117 sends the read signal "read" to the memory 
circuit 116, as shown in (g). When this read signal "read" becomes high, the 
memory circuit 116 reads out a packet as shown in (h). The packet thus 
read out forms the transport stream TS m . In the example shown in FIG. 17, 
the packet (n+2) has disappeared between a packet (n+1) and a packet (n+3) 
as represented as Loss in (h), however there are no blank time slot. In the 
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present embodiment, the overhead of the private packet can thus be 
eliminated. 

Next, with reference to FIG. 18, operation timing of the 
multiplexing control circuits 103 and 104 including the overflow avoiding 
operation of the memory circuit 116 will now be described. In FIG. 18, (a) 
represents the request signal REQ1, (b) packets in the transport stream TS d , 
(c) the noncoincidence signal "mismatch," (d) the write signal "write," (e) 
packets outputted from the delay circuit 115, (f) the wait signal "wait," (g) 
the number of packets monitored by the memory control circuit 117, (h) the 
request signal REQ2, (i) the read signal "read," and (j) packets in the 
transport stream TS m . 

In FIG. 18, write operation up to the packet (n+3) is the same as 
that of the example shown in FIG. 17. It is assumed in this example that 
the memory control circuit 117 conducts control so as to accumulate up to K 
packets in total in the memory circuit 116, and it is provided with a packet 
counter for counting packets to grasp the accumulation of packets. FIG. 18 
(g) represents the count value of this packet counter. The packet counter 
counts up by one at the start (rising edge) of the write signal "write" supplied 
from the memory control circuit 117, and counts down by one at the end 
(falling edge) of the read signal "read" supplied from the memory control 
circuit 117. It is assumed in the example shown in FIG. 18 that K-3 packets 
have been accumulated in the memory circuit 116 at first. As shown in (g), 
therefore, the value of the packet counter is K-3 at first. 

When writing a packet (n-1) into the memory circuit 116 is started, 
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the packet counter counts up by one, and the value of the packet counter 
becomes K-2. When writing a packet (n) into the memory circuit 116 is 
started, then the packet counter counts up by one, and the value of the 
packet counter becomes K-l. If reading a packet (m) from the memory 
circuit 116 is finished as shown in (j), the packet counter counts down by one, 
and the value of the packet counter becomes K-2. Subsequently, when 
writing a packet (n+1) into the memory circuit 116 is started, the packet 
counter counts up by one, and the value of the packet counter becomes K-l. 
Subsequently, since a packet (n+2) is a private packet, it is not written into 
the memory circuit 116. Therefore, the value of the packet counter remains 
K-l. Subsequently, when writing a packet (n+3) into the memory circuit 
116 is started, the packet counter counts up by one, and the value of the 
packet counter becomes K. 

When the value of the packet counter becomes K, the memory 
control circuit 117 sends the wait signal "wait", as shown in FIG. 18(f). 
Upon receiving the wait signal "wait," the rate control circuit 112 does not 
issue the request signal REQ1, even at timing "stopl" for issuing the request 
signal REQ1, as shown in (a). Subsequently, when reading a packet (m+1) 
from the memory circuit 116 is finished, the packet counter counts down by 
one, and the value of the packet counter becomes k-l. When the value of 
the packet counter becomes K-l or less, the memory control circuit 117 
suspends sending the wait signal "wait." 

Upon suspension of the wait signal "wait" sending, the rate control 
circuit 112 sends out the request signal REQ1 again. Upon start of writing 
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a packet (n+4) into the memory circuit 116 by the request signal REQ1, the 
packet counter counts up by one, and the value of the packet counter 
becomes K again. From the memory control circuit 117, the wait signal 
"wait" is sent out again. Thereafter, while the wait signal "wait" is being 
sent, the request signal REQ1 is not issued even when timing for issuing the 
request signal REQ1, such as "stop2", "stop3", and "stop4", is reached. 

Then, if reading a packet (m+2) from the memory circuit 116 is 
finished, the packet counter counts down by one, and the value of the packet 
counter becomes K-l. The memory control circuit 117 suspends sending the 
wait signal "wait." If K is set to a value smaller than the number of packets 
causing the overflow of the memory circuit 116 according to such control, 
packets more than K packets are not accumulated in the memory circuit 116 
owing to such control. As a result, the memory circuit 116 does not 
overflow. 

Next, with reference to a flow chart of FIG. 19, operation of the 
encoding device 2; relating to the statistical multiplexing will now be 
described. In this operation , first of all, each encoding device % i is 
initialized to a requisite state (step S101). Thereafter, on the basis of the 
intra AC and the ME residual, each encoding device 2 4 derives the encoding 
difficulty D; concerning an image to be encoded by using the encoding 
difficulty calculation unit 42 of the video encoder 10. Then, each encoding 
device % determines whether the encoding difficulties of one frame have been 
calculated (step S102) or not. Each encoding device % waits until the 
calculation of the encoding difficulties corresponding to one frame is finished 
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(step S102; N). When the calculation of the encoding difficulties 
corresponding to one frame has been finished (step S102; Y), each encoding 
device adds a PID or an encoding device identification number of each 
encoding device 2i (step S103), forms a private packet (step S104), and 
transmits the private packet to the multiplexer 4 via the same transmission 
channel 6 { as that of the video packet and the audio packet (step S105). 

Next, the CPU 65 of each encoding device 2 S determines whether the 
target bit rate has been received from the statistical multiplexing computer 3 
(step S106) or not. The CPU 65 waits until the target bit rate data is 
received (step S106; N). When the target bit rate data has been received 
(step S106; Y), the CPU 65 extracts the target bit rate Rate; pertinent to its 
own encoding device (step S107). The extracted target bit rate Ratej is set 
in the quantization index determining unit 45 of the video encoder 10 via the 
interface 72 (step S108). The quantization index determining unit 45 
determines the quantization index corresponding to the quantization 
characteristic value in the quantization circuit 33 so as to attain the set 
target bit rate Rate;, and sends it to the quantization circuit 33. In response 
to this, encoding of a picture j is conducted (step S109). By the way, the 
picture j means a picture to be encoded now. 

Upon termination of the encoding of the picture j, j+1 is adopted as 
new j for the next picture processing (step SI 10), and it is determined 
whether the encoding should be finished (step Sill) or not. In case that 
encoding should be continued (step Sill; N), the processing returns to the 
step S102. In case that encoding should be finished (step Sill; Y), the 
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operation shown in FIG. 19 is finished. 

Next, with reference to a flow chart of FIG. 20, an example of 
operation of the statistical multiplexing computer 3 will now be described. 
In this example, the statistical multiplexing computer 3 can be switched over 
from one of an image quality preference mode and an average bit rate 
preference mode to another, channel by channel. In the average bit rate 
preference mode, the statistical multiplexing computer 3 accepts setting of 
an average bit rate (Avg), a lower limit bit rate (Min), and an upper limit bit 
rate (Max), distributes the average bit rate thus accepted to allocate a 
remaining bit rate equally to channels of image quality preference. As for 
channels of the image quality preference mode, the statistical multiplexing 
computer 3 does not accept setting of an average bit rate (Avg), a lower limit 
bit rate (Min), and an upper limit bit rate (Max). 

In the operation shown in FIG. 20, the statistical multiplexing 
computer 3 first executes computation processing of the following expression 
(6) for each channel with respect to pictures of a predetermined number of 
frames subsequent to a picture to be encoded, and thereby calculates a time 
averaged encoding difficulties AD i of each channel (step S201). Unit time 
for deriving the time averaged value is set to 1 second. In the expression (6), 
Z means the sum total with respect to N pictures ranging from the picture 
to be encoded to a picture corresponding to the unit time for deriving the 
time averaged value. Furthermore, j represents a number of an encoding 
device, D d represents an encoding difficulty of each picture, and picture_rate 
represents a picture rate. 
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Adj= ZDj x picture_rate / N —(6) 

From the time averaged encoding difficulty AD; thus calculated, the 
statistical multiplexing computer 3 then prescribes a rate setting function 
for each channel, and distributes a temporary bit rate to each channel 
according to the function (step S202). 

For channels of the average bit rate preference mode, the statistical 
multiplexing computer 3 prescribes a function, as shown in FIG. 21 so that 
the time averaged encoding difficulty ADj of each channel may correspond to 
the average bit rate set for each channel and a proportional relation may be 
formed between the lower limit value and the upper limit value in bit rate. 
Furthermore, by using the encoding difficulty Dj of each picture, a bit rate 
determined by each function thus set is detected and the bit rate thus 
detected is set to each of channels of the average bit rate preference. 

In other words, the statistical multiplexing computer 3 calculates a 
temporary bit rate Tmpjlatej for a channel of the average bit rate preference 
by using the computation processing of the following expression (7). In the 
expression (7), the time averaged encoding difficulty ADj is represented as 
D avg . Furthermore, Rate ave , Rate min , and Rate max represent an average bit 
rate, a lower limit bit rate, and an upper limit bit rate, respectively. 
Furthermore, min(A, B) is a function for selecting one having a value smaller 
than a value A and a value B. 
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Tmp_Ratej = min {Rate min + (Rate ave - Rate min ) / D avg x Dj, Rate ma J 

-(7) 

On the other hand, for channels of the image quality preference, the 
statistical multiplexing computer 3 calculates a channel averaged encoding 
difficulty 2 ADj/N q from the time averaged encoding difficulty AD^ calculated 
at the step S201. Here, N q is the number of channels of the image quality 

J; preference. In addition, the statistical multiplexing computer 3 prescribes a 

lid 

£2 function common to the channels of the image quality preference, as shown 

iD 

m in FIG. 22 so that the channel averaged encoding difficulty ZAD/N q 

m 

y> (represented as Mean Davg in FIG. 22) may correspond to an average bit 

P rate of channels (represented as Avg in FIG. 22) and a proportional relation 

may be formed between a lower limit value and an upper limit value in bit 
"% rate. Here, the average bit rate in the channels of the image quality 

preference is obtained by subtracting the average bit rate set in channels of 
the average bit rate preference from the total bit rate, and distributes a 
resultant remained bit rate equally to the channels of the image quality 
preference. Furthermore, the minimum value and the maximum value are 
preset values. 

Next, the statistical multiplexing computer 3 detects a bit rate 
determined by the function set as described above according to the encoding 
difficulty Dj of each picture, for each of the channels of image quality 
preference, and sets the bit rate to each of the channels of image quality 
preference. 
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Upon thus setting temporary bit rates, the statistical multiplexing 
computer 3 executes temporary bit rate correction processing (step S203). 
By this correction processing, the statistical multiplexing computer 3 sets 
the total bit rate so that the total bit rate may become the bit rate assigned 
to the statistical multiplexing system 1. 

FIG. 23 shows temporary bit rate correction processing. By 
determining whether a relation of the following expression (8) holds true in 
this processing, the statistical multiplexing computer 3 determines whether 
a sum total bit rate Sum_Tmp_Rate obtained from the temporary bit rate 
Tmp_Rate j of each channel detected at the step S202 exceeds the bit rate 
(Total_Rate) assigned to the statistical multiplexing system 1 (step S301). 

Sum_Tmp_Rate > Total_Rate -(8) 

When a negative result is obtained (N), there is still a margin. If 
transmission is conducted at this temporary bit rate, null bits having no 
meaning are transmitted. Therefore, the statistical multiplexing computer 
3 proceeds to step S302. Here, the statistical multiplexing computer 3 
detects such channels that the bit rate is not limited by the upper limit value 
Max of the bit rate . Furthermore, by using computation processing of the 
following expression (9), the statistical multiplexing computer 3 
proportionally allocate the margin bit rate to the detected channels, and 
derives a bit rate Rate; (step S302). 
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Rate; = Tmp.Ratej x (Total_Rate / Sum_Tmp_Rate) ...(9) 

In the expression (9), Total_Rate is a remaining bit rate obtained by 
subtracting the bit rate of the channels limited in bit rate by the upper limit 
bit rate value Max from the bit rate Total_Rate assigned to the statistical 
multiplexing system 1. Sum_Tmp Rate is the sum total of the temporary 
rates Tmp_Ratej temporarily set for channels which are not limited in bit 
rate by the upper limit bit rate value Max. 

Bit rates are thus set. By determining whether the following 
expression (10) holds true in each channel, the statistical multiplexing 
computer 3 determines whether there is a channel having a bit rate 
exceeding the upper limit bit rate value Max (Rate (Max)) in the channels 
having bit rates thus set (step S303). 

Rate 3 > Rate(Max) -( 10 ) 

If a bit rate exceeding the upper limit bit rate value Max 
(Rate(Max)) exists (Y), the statistical multiplexing computer 3 sets (clips) the 
bit rate Ratej of the channel having a bit rate exceeding the upper limit bit 
rate value Max (Rate(Max)) to the upper limit value (step S304), and 
thereafter returns to the step S302. 

As a result, the statistical multiplexing computer 3 repeats the 
processing procedure of steps S302 - S303 - S304 - S302 as occasion demands. 
In such a range as not to exceed the upper limit value in each channel, the 
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statistical multiplexing computer 3 thus conducts allocation of a remaining 
bit rate after temporary setting, and sets the bit rate of each channel. 
Further, upon completion of this setting, a negative result (N) is obtained at 
the step S303, and the temporary bit rate correction processing is finished. 

On the other hand, when the sum total bit rate Sum_Tmp Rate 
obtained from the temporary bit rate TmpJRatej of each channel detected at 
the step S202 exceeds the bit rate (Total_Rate) assigned to the statistical 
multiplexing system 1 (step S301; Y), the statistical multiplexing computer 3 
proceeds to step S305. 

Here, the statistical multiplexing computer 3 detects such channels 
that the bit rate is not limited by the lower limit value Min of the bit rate. 
Furthermore, by using computation processing of the following expression 
(11), the statistical multiplexing computer 3 proportionally allocate the extra 
bit rate exceeding the bit rate Total_Rate assigned to the statistical 
multiplexing system 1 to the detected channels, and derives a bit rate Rate, 
(step S305). 

Rat^ = TmpJEtatej x (Total_Rate / Sum_Tmp_Rate) ...(11) 

In the expression (11), Total_Rate is a remaining bit rate obtained 
by subtracting the bit rate of the channels limited in bit rate by the lower 
limit bit rate value Min from the bit rate Total_Rate assigned to the 
statistical multiplexing system 1. Sum_Tmp_Rate is the sum total of the 
temporary rates TmpJEtatej temporarily set for channels which are not 
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limited in bit rate by the lower limit bit rate value Min. 

Bit rates are thus set. By determining whether the following 
expression (12) holds true in each channel, the statistical multiplexing 
computer 3 determines whether there is a channel having a bit rate less than 
the lower limit bit rate value Min (Rate(Min)) in the channels having bit 
rates thus set (step S306). 

Ratej < Rate(Min) -(12) 

When a bit rate less than the lower limit bit rate value Min 
(Rate(Min)) exists (Y), the statistical multiplexing computer 3 sets (clips) the 
bit rate Rate, of the channel having a bit rate less than the lower limit bit 
rate value Min (Rate(Min)) to the lower limit value (step S307), and then 
returns to the step S305. 

As a result, the statistical multiplexing computer 3 repeats the 
processing procedure of steps S305 - S306 - S307 - S305 as occasion demands. 
In such a range as not to be less the lower limit value in each channel, the 
statistical multiplexing computer 3 thus conducts allocation of the bit rate 
shortage after temporary setting, and sets the bit rate of each channel. 
Further, upon completion of this setting, a negative result (N) is obtained at 
the step S306, and the temporary bit rate correction processing is finished. 

Upon thus correcting the temporary bit rates and setting the bit 
rate of each channel, the statistical multiplexing computer 3 proceeds to step 
S204 (FIG. 20) and notifies each encoding device 2 ; of the bit rate thus 



- 65 - 

calculated. Subsequently, by incrementing the variable j, the statistical 
multiplexing computer 3 sets the picture to be thus calculated in bit rate to 
the next picture (step S205). 

Subsequently, by updating the time averaged encoding difficulty 
D avg and the channel averaged encoding difficulty Mean_D avg> the statistical 
multiplexing computer 3 updates the reference value prescribing the 
functions which have been used to calculate the temporary bit rates (step 
S206). 

As for the channels of the average bit rate preference, the statistical 
multiplexing computer 3 then determines whether the picture is the last 
picture of the GOP. In the case of the last picture, the statistical 
multiplexing computer 3 executes computation processing of the following 
expression (13) and thereby updates the time averaged encoding difficulty 
value D avr 

D avg = {(k-l)xD a ^ + Ad i }/k ...(13) 

The encoding difficulty D avg on the right side is the encoding 
difficulty of the calculation reference of one GOP before. Furthermore, k is 
a weighting coefficient, and a sufficiently large integer is used as k. As a 
result, the statistical multiplexing computer 3 changes the encoding 
difficulty D avg of this calculation reference by using a predetermined time 
constant according to a change of the encoding difficulty Dj. For example, 
the statistical multiplexing computer 3 changes the time averaged function 
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used until then as represented by a broken line in FIG. 24 to a time averaged 
function as represented by a solid line. 

On the other hand, as for channels of the image quality preference, 
the statistical multiplexing computer 3 sets any one of channels as a channel 
of update reference, and determines whether the picture is the last picture of 
the GOP in the channel of update reference. In the case of the last picture, 
the statistical multiplexing computer 3 executes computation processing of 
the following expression (14) and thereby updates the channel averaged 
encoding difficulty Mean_D avg . 

Mean_D avg = {(k - 1) x Mean_D avg + Z Ad, / NJ / k ...(14) 

Nq is the number of channels of the image quality preference. The 
encoding difficulty value MeanJDavg on the right side is the encoding 
difficulty of the calculation reference of one GOP before. As a result, the 
statistical multiplexing computer 3 derives the channel average EAD/Nq of 
the encoding difficulty of channels of the image quality preference, 
conducts weighted averaging thereof, and thereby updates the encoding 
difficulty Mean_D avg of the calculation reference. For example, the 
statistical multiplexing computer 3 changes the channel averaged function 
used until then as represented by a broken line in FIG. 25 to a channel 
averaged function as represented by a solid line. 

Upon thus updating the encoding difficulty values D avg and 
Mean_D avg , the statistical multiplexing computer 3 determines whether the 
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program is terminated (step S207). When a negative result (N) is obtained, 
the control returns to step S201. While updating the time averaged 
function and channel averaged function by taking a GOP as the unit, the 
statistical multiplexing computer 3 thus calculates the bit rates sequentially 
by taking a picture as the unit, and notifies each encoding device 2 { of the 
calculated bit rate. Further, upon completion of a series of processing, an 
affirmative result (Y) is obtained at the step S207, and the statistical 
multiplexing computer 3 finishes the control operation using the statistical 
multiplexing. 

As for channels of the average bit rate preference, the statistical 
multiplexing computer 3 sets the target bit rate so that the bit rate may 
change according to the encoding difficulty, before and after the average bit 
rate set for each channel. As for channels of the image quality preference, 
the statistical multiplexing computer 3 sets the target bit rate so that the 
total bit rate may change according to the total encoding difficulty, before 
and after the remaining bit rate obtained by subtracting the average bit rate 
of the channels of the average bit rate preference from the total bit rate. 

As described above, according to the present embodiment, the 
encoding difficulties D ; generated by the respective encoding devices 2, are 
transmitted to the multiplexer 4 via the same transmission channels 6 t as 
the video data and the audio data by utilizing the private packet of the 
transport stream in the MPEG system, and multiplexed by the multiplexer 4. 
Multiplexed transport stream TS d is thus generated and transmitted to the 
statistical multiplexing computer 3. The statistical multiplexing computer 
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3 extracts the encoding difficulties D t of the respective encoding devices 2 { 

from the transport stream TS d supplied from the multiplexer 4. On the 

basis of the encoding difficulties Dj, the statistical multiplexing computer 3 

calculates the target bit rate Rat^ of the respective encoding devices 2 t . As 

a result, it becomes possible to transmit a large number of encoding 

difficulties to the statistical multiplexing computer 3. 

Also, according to the present embodiment, the multiplexer 4 

O conducts multiplexing processing on data supplied from the encoding devices 

W 2j at the first rate Rl larger than the data transmission rate on the 

transmission channel of the subsequent stage, and outputs the transport 

U stream TS d including the private packets for statistical multiplexing to the 

h statistical multiplexing computer 3. In addition, the multiplexer 4 conducts 

ft ' , 

I* multiplexing processing on data obtained by removing the data for statistical 

* multiplexing from the data outputted from the respective encoding devices 2 i; 

at a second rate R2 which is equal to the transmission rate on the 

transmission channel of the subsequent stage, and outputs the transport 

stream TS m which does not include the private packets for statistical 

multiplexing to the transmission channel of the subsequent stage. As a 

result, the data sent to the transmission channel of the subsequent stage is 

not affected by the rate occupied by the private packet for conducting control 

using the statistical multiplexing. It thus becomes possible to prevent the 

data transmission efficiency from being lowered. 

Furthermore, in the present embodiment, the memory control 

circuit 117 in the multiplexing control circuit 103 controls the rate control 
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circuit 112 so as to prevent the overflow of the memory circuit 116. No 
matter how the rate Rl may be greater than the rate R2, therefore, the 
memory circuit 116 does not overflow, but the normal operation becomes 
possible. 

The present invention is not limited to the above described 
embodiment. For example, in the embodiment, the statistical multiplexing 
computer 3 takes out the private packet from the transport stream TS d . 
Alternatively, the multiplexer 4 may take out only the private packets from 
the transport stream TS d , and output the private packets to the statistical 
multiplexing computer 3. 

Furthermore, according to the present invention, not only the low 
rate data such as the private packet used for statistical multiplexing, but 
also a higher rate data can be sent from the encoding device 2 f to the 
multiplexer 4. The data can be sent from the multiplexer 4 and used for 
various kinds of control. 

As described above, according to the video data multiplexing device 
or the video data multiplexing control method of the present invention, 
statistical multiplexing data is generated and outputted on the same 
transmission channel as the encoded stream are transmitted, by the 
encoding means. The encoded stream and the statistical multiplexing data 
are acquired from each encoding means, and multiplexed and outputted by 
the multiplexing means. The statistical multiplexing data of each encoding 
means is acquired from output of the multiplexing means, and control using 
statistical multiplexing is conducted on each encoding means on the basis of 
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the statistical multiplexing data, by the encoding control means. This 
results in an effect that the statistical multiplexing data required for control 
using the statistical multiplexing can be efficiently transmitted. 

According to the video data multiplexing device or the video data 
multiplexing control method of the present invention, the multiplexing 
means removes the statistical multiplexing data out of data obtained by 
multiplexing the encoded stream and the statistical multiplexing data 
supplied from the respective encoding means, and outputs resultant data to 
a transmission channel of a subsequent stage. This results in an effect that 
data transmission can be conducted without uselessness with respect to 
original multiplexed data to be outputted to the subsequent stage. 

Furthermore, according to the video data multiplexing device or the 
video data multiplexing control method of the present invention, the packet 
of the statistical multiplexing data includes identification data and data for 
rejection detection. This result in an effect that it becomes possible to 
detect whether there has been rejection of a packet for statistical 
multiplexing data and prevent a system failure due to rejection of a packet 
for statistical multiplexing data. 

Furthermore, according to the video data multiplexing device or the 
video data multiplexing control method of the present invention, each 
encoding means generates statistical multiplexing data required for control 
using statistical multiplexing, and outputs the generated data on the same 
transmission channel as the encoded stream are transmitted. Furthermore, 
the multiplexing means acquires the encoded stream and the statistical 
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multiplexing data from the respective encoding means via the transmission 
channels, conducts multiplexing processing on the encoded streams and the 
statistical multiplexing data at a first rate greater than a data transmission 
rate on a transmission channel of a subsequent stage, and outputs first data 
including the statistical multiplexing data. In addition, the multiplexing 
means conducts multiplexing processing on data obtained by removing the 
statistical multiplexing data from the data outputted from the respective 
encoding means, at a second rate equal to a data transmission rate on the 
transmission channel of the subsequent stage, and outputs second data 
which does not include the statistical multiplexing data to the transmission 
channel of the subsequent stage. An encoding control means acquires the 
statistical multiplexing data of each encoding means from the first data 
outputted from the multiplexing means, and conducts control using 
statistical multiplexing on each encoding means on the basis of the 
statistical multiplexing data. This results in an effect that the statistical 
multiplexing data required for the control by the statistical multiplexing can 
be transmitted efficiently, and the second data sent to the transmission 
channel of the subsequent stage is not affected by the statistical 
multiplexing data, thus the data transmission efficiency being prevented 
from being lowered. 

Furthermore, according to the encoded stream multiplexing device 
or the encoded stream multiplexing method according to the present 
invention, a plurality of transport streams including the video transport 
stream packets and private transport stream packets, which include the 
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encoding difficulty information indicating the encoding difficulty at the time 
of encoding video data of a plurality of channels, are multiplexed. A 
multiplexed transport stream is thus generated. A private transport 
stream packet included in the multiplexed transport stream is extracted. 
On the basis of the encoding difficulty information described in the extracted 
private transport stream packets, the target encoding rates respectively 
corresponding to the plurality of channels are computed. This results in an 
effect that the encoding difficulty information can be transmitted efficiently. 

Furthermore, according to the encoding device or the encoding 
method according to the present invention, a plurality of encoded video 
streams generated by encoding the video data of the plurality of channels are 
outputted as video transport stream packets. In addition, encoding 
difficulty information indicating encoding difficulty in encoding video data of 
the plurality of channels is outputted as private transport stream packets. 
On the basis of the encoding difficulty information described in the outputted 
private transport stream packets, the target encoding rates respectively 
corresponding to the plurality of channels are computed, and thereby rates of 
the encoded streams are controlled. This results in an effect that the 
encoding difficulty information can be transmitted efficiently. 

From the description heretofore made, it is apparent that various 
aspects and modifications of the present invention can be implemented. In 
the scope of the following claims, therefore, the present invention can be 
implemented in aspects other than the aspect heretofore described in detail. 
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What is claimed is: 

1. A video data multiplexing device comprising: 

a plurality of encoding means for encoding program data 
respectively including video data, outputting resultant encoded streams, 
generating statistical multiplexing data required for control using statistical 
multiplexing, and outputting the generated data on the same transmission 

% channels as the encoded streams are transmitted; 

W 

y, multiplexing means for acquiring the encoded streams and the 

m 

ffi statistical multiplexing data from the respective encoding means via the 

m 

H transmission channels, and multiplexing and outputting them; and 

encoding control means for acquiring the statistical multiplexing 

m 

data of each encoding means from output of the multiplexing means, and 
% 1 conducting control using statistical multiplexing on each encoding means on 

the basis of the statistical multiplexing data. 

2. A video data multiplexing device according to claim 1, wherein the 
encoding means forms the encoded stream and the statistical multiplexing 
data respectively as packets and outputs the packets. 

3. A video data multiplexing device according to claim 1, wherein the 
multiplexing means includes statistical multiplexing data removing means 
for removing the statistical multiplexing data out of data obtained by 
multiplexing the encoded streams and the statistical multiplexing data 
supplied from the respective encoding means, and outputting resultant data 
to a transmission channel of a subsequent stage, and the multiplexing means 
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outputs the data including the statistical multiplexing data to the encoding 
control means without passing through the statistical multiplexing data 
removing means. 

4. A video data multiplexing device according to claim 2, wherein the packet 
of the statistical multiplexing data includes identification data for 
identifying from which encoding means the statistical multiplexing data is 
supplied. 

5. A video data multiplexing device according to claim 4, wherein the packet 
of the statistical multiplexing data further includes data for rejection 
detection used for detecting whether packet rejection is present or not. 

6. A video data multiplexing control method used in a video data 
multiplexing device including a plurality of encoding means for encoding 
program data respectively including video data and outputting encoded 
streams, multiplexing means for multiplexing the encoded streams 
outputted from controlling each of the encoding means, and encoding control 
means for controlling each of the encoding means, wherein control using 
statistical multiplexing is conducted on each of the encoding means by the 
encoding control means, comprising: 

a statistical multiplexing data output procedure in the encoding 
means for generating statistical multiplexing data required for control using 
statistical multiplexing, and outputting the generated data on the same 
transmission channels as the encoded streams are transmitted; 

a multiplexing procedure in the multiplexing means for acquiring 
the encoded streams and the statistical multiplexing data from the 
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respective encoding means via the transmission channels, and multiplexing 
and outputting them; and 

an encoding control procedure in the encoding control means for 
acquiring the statistical multiplexing data of the respective encoding means 
from output of the multiplexing means, and conducting control using 
statistical multiplexing on the respective encoding means on the basis of the 
statistical multiplexing data. 

7. A video data multiplexing control method according to claim 6, wherein in 
the statistical multiplexing data output procedure, the statistical 
multiplexing data is formed as a packet and outputted. 

8. A video data multiplexing control method according to claim 6, wherein 
the video data multiplexing control method further includes a statistical 
multiplexing data removing procedure for removing the statistical 
multiplexing data out of data multiplexed by the multiplexing procedure and 
outputting resultant data to a transmission channel of a subsequent stage, 
in the multiplexing means. 

9. A video data multiplexing control method according to claim 7, wherein 
the packet of the statistical multiplexing data includes identification data for 
identifying from which encoding means the statistical multiplexing data is 
supplied. 

10. A video data multiplexing control method according to claim 9, wherein 
the packet of the statistical multiplexing data further includes data for 
rejection detection used for detecting whether packet rejection is present or 
not. 
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11. A video data multiplexing device comprising: 

a plurality of encoding means for encoding program data 
respectively including video data, outputting resultant encoded streams, 
generating statistical multiplexing data required for control using statistical 
multiplexing, and outputting the generated data on the same transmission 
channels as the encoded streams; 

multiplexing means for acquiring the encoded streams and the 
statistical multiplexing data from the respective encoding means via the 
transmission channels, conducting multiplexing processing on the encoded 
streams and the statistical multiplexing data at a first rate greater than a 
data transmission rate on a transmission channel of a subsequent stage, 
outputting first data including the statistical multiplexing data, conducting 
multiplexing processing on data obtained by removing the statistical 
multiplexing data from the data outputted from the respective encoding 
means, at a second rate equal to a data transmission rate on the 
transmission channel of the subsequent stage, and outputting second data 
which does not include the statistical multiplexing data to the transmission 
channel of the subsequent stage; and 

encoding control means for acquiring the statistical multiplexing 
data of the respective encoding means from the first data outputted from the 
multiplexing means, and conducting control using statistical multiplexing on 
the respective encoding means on the basis of the statistical multiplexing 
data. 

12. A video data multiplexing device according to claim 11, wherein the 
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encoding means forms the encoded streams and the statistical multiplexing 
data respectively as packets and outputs the packets. 
13. A video data multiplexing device according to claim 11, wherein the 
multiplexing means includes a multiplexing unit for acquiring the encoded 
streams and the statistical multiplexing data from the respective encoding 
means via the transmission channels, and multiplexing them, a first 
multiplexing control unit for controlling the multiplexing unit so that the 
first data may be outputted from the multiplexing unit at the first rate, and 
holding data obtained by removing the statistical multiplexing data from the 
first data outputted from the multiplexing unit, and a second multiplexing 
control unit for controlling the first multiplexing control unit so that the data 
held by the first multiplexing control unit may be outputted to the 
transmission channel of the subsequent stage at the second rate as the 
second data. 

14. A video data multiplexing control method used in a video data 
multiplexing device including a plurality of encoding means for encoding 
program data respectively including video data and outputting encoded 
streams, multiplexing means for multiplexing the encoded streams 
outputted from controlling each of the encoding means, and encoding control 
means for controlling each of the encoding means, wherein control using 
statistical multiplexing is conducted on each of the encoding means by the 
encoding control means, comprising: 

a statistical multiplexing data output procedure in the encoding 
means for generating statistical multiplexing data required for control using 
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statistical multiplexing, and outputting the generated data on the same 
transmission channel as the encoded streams are transmitted; 

a multiplexing procedure in the multiplexing means for acquiring 
the encoded streams and the statistical multiplexing data from the 
respective encoding means via the transmission channels, conducting 
multiplexing processing on the encoded streams and the statistical 
multiplexing data at a first rate greater than a data transmission rate on a 
transmission channel of a subsequent stage, outputting first data including 
the statistical multiplexing data, conducting multiplexing processing on data 
obtained by removing the statistical multiplexing data from the data 
outputted from the respective encoding means, at a second rate equal to a 
data transmission rate on the transmission channel of the subsequent stage, 
and outputting second data which does not include the statistical 
multiplexing data to the transmission channel of the subsequent stage; and 

an encoding control procedure in the encoding control means for 
acquiring the statistical multiplexing data of the respective encoding means 
from the first data outputted from the multiplexing means, and conducting 
control using statistical multiplexing on the respective encoding means on 
the basis of the statistical multiplexing data. 

15. A video data multiplexing control method according to claim 14, wherein 
in the statistical multiplexing data output procedure, the statistical 
multiplexing data is formed as packets and outputted. 

16. An encoded stream multiplexing device for multiplexing encoded streams, 
comprising: 
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a plurality of encoding means for respectively encoding video data of 
a plurality of channels on the basis of target encoding rates and outputting 
encoded video streams; 

encoding control means for computing the target encoding rates 
supplied to the plurality of encoding means for the respective channels; and 
multiplexing means for multiplexing a plurality of encoded streams 
respectively outputted from the plurality of encoding means, 

wherein the plurality of encoding means output the encoded video 
streams as video transport stream packets, and output encoding difficulty 
information indicating encoding difficulties in encoding video data of the 
plurality of channels as private transport stream packets, 

the multiplexing means includes a multiplexing circuit for receiving 
a plurality of transport streams including the video transport stream packets 
and the private transport stream packets respectively from the plurality of 
encoding means, multiplexing the plurality of transport streams, and 
thereby generating a multiplexed transport stream, and 

the encoding control means receives the multiplexed transport 
stream from the multiplexing means, extracts the private transport stream 
packets included in the multiplexed transport stream, and computes the 
target encoding rates respectively corresponding to the plurality of channels 
on the basis of the encoding difficulty information described in the extracted 
private transport stream packets. 

17. An encoded stream multiplexing device according to claim 16, wherein 
the multiplexing means further includes private packet removing means for 
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removing the private transport stream packets from the multiplexed 
transport stream, and generating a multiplexed transport stream which does 
not include the private transport stream packets. 

18. An encoded stream multiplexing device according to claim 17, wherein in 
a header of each transport stream packet included in the transport stream, a 
packet identifier representing a kind of data included in the transport 
stream packet is described, and 

the private packet removing means compares a packet identifier of 
each transport stream packet included in the multiplexed transport stream 
with a unique packet identifier set for each private transport stream packet, 
and thereby removes the private transport stream packet from the 
multiplexed transport stream. 

19. An encoded stream multiplexing device according to claim 18, wherein 
the private packet removing means controls output timing of each transport 
stream packet included in the multiplexed transport stream with the private 
transport stream packets removed therefrom so as to eliminate vacant time 
slots generated by removing the private transport stream packets. 

20. An encoded stream multiplexing device according to claim 16, wherein 
the encoding control means computes temporary encoding rates respectively 
corresponding to the plurality of channels from the plurality of pieces of 
encoding difficulty information transmitted by the private transport stream 
packets, and computes the target encoding rates from the temporary 
encoding rates so as to make a sum total of the temporary encoding rates 
equal to or less than a transmission rate. 
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21. An encoded stream multiplexing device for multiplexing encoded streams, 
comprising: 

a plurality of encoding means for respectively encoding video data of 
a plurality of channels on the basis of target encoding rates, thereby 
generating encoded video streams, outputting the encoded video streams as 
video transport stream packets, outputting encoding difficulty information 
indicating encoding difficulties in encoding video data of the plurality of 
channels as private transport stream packets; 

multiplexing means for receiving a plurality of transport streams 
including the video transport stream packets and the private transport 
stream packets respectively from the plurality of encoding means, 
multiplexing the plurality of transport streams, and thereby generating a 
multiplexed transport stream; and 

encoding control means for receiving the multiplexed transport 
stream from the multiplexing means, extracting the private transport 
stream packets included in the multiplexed transport stream, computing the 
target encoding rates respectively corresponding to the plurality of channels 
on the basis of the encoding difficulty information described in the extracted 
private transport stream packets, supplying the computed target encoding 
rates respectively to the plurality of encoding means, and thereby controlling 
rates of the encoded streams outputted from the plurality of encoding means. 
22. An encoded stream multiplexing device according to claim 21, wherein 
the multiplexing means further includes a private packet removing means 
for removing the private transport stream packets from the multiplexed 
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transport stream, and generating a multiplexed transport stream which does 
not include the private transport stream packets. 

23. An encoded stream multiplexing device according to claim 22, wherein in 
a header of each transport stream packet included in the transport stream, a 
packet identifier representing a kind of data included in the transport 
stream packet is described, and 

the private packet removing means compares the packet identifier 
of each transport stream packet included in the multiplexed transport 
stream with a unique packet identifier set for each private transport stream 
packet, and thereby removes the private transport stream packet from the 
multiplexed transport stream. 

24. An encoded stream multiplexing device according to claim 23, wherein 
the private packet removing means controls output timing of each transport 
stream packet included in the multiplexed transport stream with the private 
transport stream packets removed therefrom so as to eHminate vacant time 
slots generated by removing the private transport stream packets. 

25. An encoded stream multiplexing device according to claim 21, wherein 
the encoding control means computes temporary encoding rates respectively 
corresponding to the plurality of channels from the plurality of pieces of 
encoding difficulty information transmitted by the private transport stream 
packets, and computes the target encoding rates from the temporary 
encoding rates so as to make a sum total of the temporary encoding rates 
equal to or less than a transmission rate. 

26. An encoded stream multiplexing method for multiplexing a plurality of 
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encoded streams generated by encoding video data of a plurality of channels, 
including the steps of: 

encoding video data of the plurality of channels, generating a 
plurality of encoded streams, and calculating encoding difficulty information 
indicating encoding difficulties in encoding video data of the plurality of 
channels; 

outputting the plurality of encoded streams as video transport 
stream packets, and outputting the encoding difficulty information as 
private transport stream packets; 

respectively receiving a plurality of transport streams including the 
video transport stream packets and the private transport stream packets, 
multiplexing the plurality of transport streams, and thereby generating a 
multiplexed transport stream; and 

receiving the multiplexed transport stream, extracting the private 
transport stream packets included in the multiplexed transport stream, and 
computing the target encoding rates respectively corresponding to the 
plurality of channels on the basis of the encoding difficulty information 
described in the extracted private transport stream packets. 

27. An encoded stream multiplexing method according to claim 26, further 
comprising the step of removing the private transport stream packets from 
the multiplexed transport stream, and generating a multiplexed transport 
stream which does not include the private transport stream packets. 

28. An encoded stream multiplexing method according to claim 27, wherein 
in a header of each transport stream packet included in the transport stream, 
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a packet identifier representing a kind of data included in the transport 
stream packet is described, and 

a packet identifier of each transport stream packet included in the 
multiplexed transport stream is compared with a unique packet identifier 
set for each private transport stream packet, and thereby the private 
transport stream packets are removed from the multiplexed transport 
stream. 

29. An encoded stream multiplexing method according to claim 28, wherein 
output timing of each transport stream packet included in the multiplexed 
transport stream with the private transport stream packets removed 
therefrom is controlled so as to eliminate vacant time slots generated by 
removing the private transport stream packets. 

30. An encoded stream multiplexing method according to claim 26, wherein 
temporary encoding rates respectively corresponding to the plurality of 
channels are computed from the plurality of pieces of encoding difficulty 
information transmitted by the private transport stream packets, and the 
target encoding rates from the temporary encoding rates are computed so as 
to make a sum total of the temporary encoding rates equal to or less than a 
transmission rate. 

31. An encoded stream multiplexing method for multiplexing a plurality of 
encoded streams, comprising: 

a plurality of encoding steps for respectively encoding video data of 
a plurality of channels on the basis of target encoding rates, thereby 
generating encoded video streams, outputting the encoded video streams as 
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video transport stream packets, outputting encoding difficulty information 
indicating encoding difficulties in encoding video data of the plurality of 
channels as private transport stream packets; 

a multiplexing step for receiving a plurality of transport streams 
including the video transport stream packets and the private transport 
stream packets respectively from the plurality of encoding steps, 
multiplexing the plurality of transport streams, and thereby generating a 
multiplexed transport stream; and 

an encoding control step for receiving the multiplexed transport 
stream from the multiplexing step, extracting the private transport stream 
packets included in the multiplexed transport stream, computing the target 
encoding rates respectively corresponding to the plurality of channels on the 
basis of the encoding difficulty information described in the extracted private 
transport stream packets, supplying the computed target encoding rates 
respectively to the plurality of encoding steps, and thereby controlling rates 
of the encoded streams outputted from the plurality of encoding steps. 

32. An encoded stream multiplexing method according to claim 31, wherein 
the multiplexing step further includes a private packet removing step for 
removing the private transport stream packets from the multiplexed 
transport stream, and generating a multiplexed transport stream which does 
not include the private transport stream packets. 

33. An encoded stream multiplexing method according to claim 32, wherein 
in a header of each transport stream packet included in the transport stream, 
a packet identifier representing a kind of data included in the transport 
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stream packet is described, and 

at the private packet removing step, a packet identifier of each 
transport stream packet included in the multiplexed transport stream is 
compared with a unique packet identifier set for each private transport 
stream packet, and thereby the private transport stream packets are 
removed from the multiplexed transport stream. 

34. An encoded stream multiplexing method according to claim 33, wherein 
at the private packet removing step, output timing of each transport stream 
packet included in the multiplexed transport stream with the private 
transport stream packets removed therefrom is controlled so as to eliminate 
vacant time slots generated by removing the private transport stream 
packets. 

35. An encoded stream multiplexing method according to claim 31, wherein 
at the encoding control step, temporary encoding rates respectively 
corresponding to the plurality of channels are computed from the plurality of 
pieces of encoding difficulty information transmitted by the private transport 
stream packets, and the target encoding rates from the temporary encoding 
rates are computed so as to make a sum total of the temporary encoding 
rates equal to or less than a transmission rate. 

36. An encoding device for encoding video data of a plurality of channels, 
comprising: 

a plurality of encoding means for respectively outputting a plurality 
of encoded video streams generated by encoding the video data of the 
plurality of channels as video transport stream packets, and outputting 
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encoding difficulty information indicating encoding difficulties in encoding 
video data of the plurality of channels as private transport stream packets; 
and 

encoding control means for computing the target encoding rates 
respectively corresponding to the plurality of channels on the basis of the 
encoding difficulty information described in the private transport stream 
packets outputted from the plurality of encoding means, supplying the 
computed target encoding rates respectively to the plurality of encoding 
means, and thereby controlling rates of the encoded streams outputted from 
the plurality of encoding means. 

37. An encoding device according to claim 36, wherein the encoding control 
means computes temporary encoding rates respectively corresponding to the 
plurality of channels from the plurality of pieces of encoding difficulty 
information transmitted by the private transport stream packets, and 
computes the target encoding rates from the temporary encoding rates so as 
to make a sum total of the temporary encoding rates equal to or less than a 
transmission rate. 

38. An encoding method for encoding video data of a plurality of channels, 
including the steps of: 

outputting a plurality of encoded video streams generated by 
encoding the video data of the plurality of channels by using a plurality of 
encoding means as video transport stream packets, and outputting encoding 
difficulty information indicating encoding difficulties in encoding video data 
of the plurality of channels as private transport stream packets; and 
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computing the target encoding rates respectively corresponding to 
the plurality of channels on the basis of the encoding difficulty information 
described in the outputted private transport stream packets, supplying the 
computed target encoding rates respectively to the plurality of encoding 
means, and thereby controlling rates of the encoded streams outputted from 
the plurality of encoding means. 

39. An encoding method according to claim 38, wherein temporary encoding 
rates respectively corresponding to the plurality of channels are respectively 
computed from the plurality of pieces of encoding difficulty information 
described in the private transport stream packets, and the target encoding 
rates are computed from the temporary encoding rates so as to make a sum 
total of the temporary encoding rates equal to or less than a transmission 
rate. 
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ABSTRACT 

The present invention makes it possible to efficiently transmit data 
for statistical multiplexing which is required for control using statistical 
multiplexing. By utilizing private packets, respective encoding devices 
transmit encoding difficulties serving as the data for statistical multiplexing 
to a multiplexer (4) via the same transmission channels as encoded video 
data and audio data are transmitted. The multiplexer (4) conducts 
multiplexing processing on data supplied from the respective encoding 
devices at a first rate Rl larger than a transmission rate on a transmission 
channel of a subsequent stage, and outputs a transport stream (TSj) 
including the private packets to a statistical multiplexing computer. In 
addition, the multiplexer (4) conducts multiplexing processing on data 
obtained by removing the private packets, at a second rate R2 which is equal 
to the transmission rate on the transmission channel of the subsequent stage, 
and outputs a transport stream (TS m ) which does not include the private 
packets to the transmission channel of the subsequent stage. 
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* DECLARATION FOR PATENT APPLICATION (JOINT OR SOLE) 

(Under 37 CFR § 1.63; with Power of Attorney) 

FROMMER LAWRENCE & HAUG LLP flh File No. 450114-4609 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name, 

I believe I am the original, first and sole inventor (if only one name is listed below) or an original, first 
and joint inventor (if plural names are listed below) of the subject matter which is claimed and for which a patent is 
sought on the invention ENTITLED: 

VIDEO DATA MULTIPLEXING DEVICE, VIDEO DATA MULTIPLEXING CONTROL METHOD, ENCODED STREAM 
MULTIPLEXING DEVICE AND METHOD, AND ENCODING D EV ICE AND METHOD 

the specification of which 

is attached hereto. 

X was filed on 15 October 1998 as International Application Serial No. PCT/JP98/04667 

with amendment(s) through (if applicable, give dates). 

I hereby state that I have reviewed and understand the contents of the above- identified specification, including 
the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose to the United States Patent and Trademark Office all information known to me 
to be material to patentability as defined in Title 37, Code of Federal Regulations, Sec. 1.56. 

I hereby claim foreign priority benefits under Title 35, United States Code, § 119 of any foreign application(s) 
for patent or inventor's certificate listed below and have also identified below any foreign application for patent or 
inventor's certificate having a filing date before that of the application on which priority is claimed: 

Prior Foreign Appl ication(s) [list additional applications on separate page]: Priority Claimed: 
Number: Country: Filed ( Day/Month/ Year ) : Yes No 

D 9-282155 Japan 15 October 1997 X 

,f\ PCT/JP98/04667 PCT 15 October 1998 X 

rT I hereby claim the benefit under Title 35, United States Code, § 120 of any United States application(s) listed 

-~ below and, insofar as the subject matter of each of the claims of this application is not disclosed in the prior United 
%0 States application in the manner provided by the first paragraph of Title 35, United States Code § 112, I acknowledge the 
fg duty to disclose to the United States Patent and Trademark Office all information known to me to be material to 

patentability as defined in Title 37, Code of Federal Regulations, Sec. 1.56, which became available between the filing 
date of the prior application and the national or PCT international filing date of this application: 
M= Prior U.S. Application(s) [list additional applications on separate page]: 

Appln. Ser. Number: Filed (Day/Month/Year): Status (patented, pending, abandoned): 



^ ~ I hereby appoint WILLIAM S. FROMMER . Registration No. 25,506. , and DENNIS M. SMID . Registration No 34., 930,, 

or their duly appointed associate, my attorneys, with full power of substitution and revocation, to prosecute this 
La application, to make alterations and amendments therein, to file continuation and divisional applications thereof, to 



receive the Patent, and to transact all business in the Patent and Trademark Office and in the Courts i 
therewith, and specify that all communications about the application are to be directed to the following correspondence 
address: 

Direct all telephone calls to: 
(212) 588-0800 
to the attention of: 

WILLIAM S. FROMMER 

I hereby declare that all statements made herein of my own knowledge are true and that all statements made on 
information and belief are believed to be true; and further that these statements were made with the knowledge that 
willful false statements and the like so made are punishable by fine or imprisonment, or both, under Section 1001 of 
Title 18 of the United States Code and that such willful false statements may jeopardize the validity of the application 
or any patent issued thereon. 

^Si S) : ^MO^. f^f^> Oate: il^lll 

nA Full name of sole or first inventor: Jo^BoMtoJaiAT^WA^^ 
\ Residence: < Kanagawa, Japan ^ 

Citizenship: '~™ 



\ Signature: r _,. — , _ „. 

yFull name of 2nd joint inventor (if any): Takao SUZUKI ^fl ^ 



Japan 

: Date: 3 -L 



/I i Residence: Kanagawa, Japan 

v Signature: 

W\Full name of 3rd joint inventor (if any): Jiw©akj_S£T0 -U \l . 

AVTResidence: Kanagawa, Japan J x 

^ Citizenship: Japan 

[Similarly list additional inventors on separate page] 

Post Office Address(es) of inventor(s): Sony Corporation 

[if all inventors have the same post office address] 7-35 Ki tashi nagawa 6-chome 

Shinagawa-Ku, Tokyo 141, Japan 

Note: In order to qualify for reduced fees available to Small Entities, each inventor and any other individual or entity 
having rights to the invention must also sign an appropriate separate "Verified Statement (Declaration) Claiming [or 
Supporting a Claim by Another for] Small Entity Status" form [e.g. for Independent Inventor, Small Business Concern, 
Nonprofit Organization, individual Non-Inventor] . 
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Citizenship: 
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